quantcore: standalone Apptainer setup for students' DGX Sparks
The portfolio data, risk and mean-CVaR optimisation service, prepared for students who run it on their own DGX Spark without root, Docker or Tailscale. - deploy/apptainer/qc.sh: install, start/stop/status/logs, cli/api/python, data import/export and Yahoo setup/update. cuOpt and quantcore run as two Apptainer instances of the user and talk over 127.0.0.1; the API and the test bench listen on 127.0.0.1 only. Builds quantcore.sif without root, or falls back to a plain Python image plus a virtualenv when the Spark allows no unprivileged builds. Detects taken ports and suggests free ones. - deploy/apptainer/quantcore.def + constraints.txt: the image, with the library versions of the verified production container pinned. - cuOpt under Apptainer: NVIDIA's image sets HOME=/opt/cuopt, read-only in a SIF, so the solver died on its first job; qc.sh gives it a writable home. - solvers: a cuOpt that disappears during a solve now falls back to HiGHS instead of answering 500 (requests' ConnectionError escaped the fallback). - client: idempotent GETs retry a dropped keep-alive connection, which made `quantcore-api update --wait` fail when a poll met uvicorn's 5 s timeout. - config: cuOpt defaults to http://127.0.0.1:5000 and `quantcore serve` to 127.0.0.1; the Docker images keep passing their own values. - docs/SPARK_APPTAINER.md: the students' guide; README, USAGE, ARCHITECTURE and SPARK_DEPLOYMENT point to it. Tested on a DGX Spark (GB10, Apptainer 1.5.4, unprivileged user) in both modes: selftest on every endpoint, optimisation on cuOpt, Yahoo update, port conflicts; 139 tests pass in the container against the live cuOpt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
herzogflorian
parent
4e3d3ae648
commit
b5da424dfe
12
.env.example
Normal file
12
.env.example
Normal file
@ -0,0 +1,12 @@
|
|||||||
|
# Copy to .env and adjust. All variables are optional; defaults are in quantcore/config.py.
|
||||||
|
QUANTCORE_DATA_DIR=./data # where the setup/download services persist everything
|
||||||
|
QUANTCORE_CUOPT_URL=http://127.0.0.1:5000 # cuOpt on this machine (Apptainer setup); a remote one: http://<host>:5000
|
||||||
|
QUANTCORE_CUOPT_TIME_LIMIT=120 # seconds, MILP time limit
|
||||||
|
QUANTCORE_BASE_CURRENCY=CHF
|
||||||
|
QUANTCORE_RISK_FREE_RATE=0.0 # annual, CHF
|
||||||
|
QUANTCORE_HISTORY_START=2004-12-01 # first date requested from Yahoo
|
||||||
|
QUANTCORE_REQUIRED_HISTORY_FROM=2005-01-31 # a stock must have prices on or before this date
|
||||||
|
QUANTCORE_N_US=500
|
||||||
|
QUANTCORE_N_EUROPE=500
|
||||||
|
QUANTCORE_LOG_LEVEL=INFO
|
||||||
|
QUANTCORE_ENABLE_UI=true # false: JSON API and /docs only, no test bench at /ui/
|
||||||
22
.gitignore
vendored
Normal file
22
.gitignore
vendored
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
# data produced by the setup / download services lives outside version control
|
||||||
|
data/
|
||||||
|
*.parquet
|
||||||
|
!quantcore/data/static/*
|
||||||
|
.env
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
.pytest_cache/
|
||||||
|
.ruff_cache/
|
||||||
|
*.egg-info/
|
||||||
|
build/
|
||||||
|
dist/
|
||||||
|
.DS_Store
|
||||||
|
scratch/
|
||||||
|
# LaTeX build files
|
||||||
|
*.aux
|
||||||
|
*.fdb_latexmk
|
||||||
|
*.fls
|
||||||
|
*.log
|
||||||
|
*.out
|
||||||
|
*.toc
|
||||||
|
*.synctex.gz
|
||||||
251
README.md
251
README.md
@ -1,3 +1,250 @@
|
|||||||
# Portfolio_Risk_Management
|
# quantcore — Portfolio Risk Measurement & Optimisation
|
||||||
|
|
||||||
The API for risk management, data downloads and portfolio optimization for AISE502
|
The **deterministic quant core** of the AI-Augmented Portfolio Intelligence Platform (AISE502):
|
||||||
|
data, risk and performance measurement, stock-price models, Black–Litterman expected returns and
|
||||||
|
mean–CVaR portfolio optimisation in whole shares, solved as a MILP on **NVIDIA cuOpt** on the DGX Spark.
|
||||||
|
|
||||||
|
Everything here is *deterministic* — same input, same output — and contains **no LLM call**. The AI
|
||||||
|
layer of the platform (news → insights → views) consumes these services through their contracts and
|
||||||
|
never computes a number itself.
|
||||||
|
|
||||||
|
| Service | Module | What it does |
|
||||||
|
|---|---|---|
|
||||||
|
| **SetupService** | `quantcore.universe` | Builds the investment universe **once**: largest 500 US (S&P 500) and 500 European (STOXX 600; EU, CH, NO, UK) stocks, Yahoo metadata (name, country, listing currency, exchange, GICS sector/industry, market cap, shares outstanding), full price history since 2004, FX rates; keeps only stocks with prices since 2005. |
|
||||||
|
| **DataDownloadService** | `quantcore.data` | Fetches the newest Yahoo prices and FX rates (incremental with dividend-adjustment splicing, or full), snapshots for reproducible demos. |
|
||||||
|
| **MarketDataService** | `quantcore.data.market_data` | Aligned adjusted closes and log-returns, daily/weekly/monthly, in CHF or local currency; market caps, cap-weighted broad market index. |
|
||||||
|
| **RiskService** | `quantcore.measurement.risk` | Volatility, correlation, covariance, VaR and CVaR (95 %, 99 %) per asset and per share-based portfolio — moving windows and exponentially weighted (EWM), Gaussian and historical, for a day / week / month, as absolute CHF amounts. |
|
||||||
|
| **PerformanceService** | `quantcore.measurement.performance` | Annualised return, volatility, Sharpe, Sortino, drawdown, Calmar, skew/kurtosis, hit ratio; rolling and EWM series; portfolios in shares. |
|
||||||
|
| **Stock-price models** | `quantcore.models.simulation` | Geometric Brownian motion (multivariate normal) and the empirical-innovation model `eps_t = Sigma_{t-1}^{-1/2} r_t`, `r(j|T) = Sigma_T^{1/2} eps_j` with the latest rolling or EWM covariance. |
|
||||||
|
| **Expected returns** | `quantcore.models.black_litterman` | Market-cap equilibrium returns (broad-market average, market weights), Black–Litterman on a 10-year weekly covariance with **Idzorek confidence** views (diagonal Ω), sentiment → view helper. |
|
||||||
|
| **OptimizationService** | `quantcore.optimization` | Mean–CVaR MILP in integer share numbers: **new portfolio** (from cash) or **rebalancing** (existing holdings, transaction costs); constraints: max 10 % per stock, max per sector, no shorts, cost/turnover limits, optional cardinality; scenario sets with variance reduction (Sobol', moment matching, market-direction importance sampling) that stay identical when risk limits, constraints, costs or views change, tail-row generation for large sets, and an exact Gaussian (mean–variance-equivalent) mode; solved on cuOpt (GPU, over HTTP) with a HiGHS fallback (CPU, compiled into SciPy — runs in-process, nothing to deploy). |
|
||||||
|
| **HTTP API** | `quantcore.api` | FastAPI app (`/docs`) exposing all of the above; setup/update as background jobs. Runs as a container on the DGX Spark next to cuOpt. |
|
||||||
|
|
||||||
|
Documentation: **[docs/USAGE.md](docs/USAGE.md) (start here if you are building on this package: the API, the
|
||||||
|
Python client, the `quantcore-api` command line, the conventions and the traps)**,
|
||||||
|
[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) (services, contracts, data flow, resilience),
|
||||||
|
[docs/METHODOLOGY.md](docs/METHODOLOGY.md) (every formula and convention),
|
||||||
|
[docs/optimization_model.pdf](docs/optimization_model.pdf) (derivation of the MILP from the Rockafellar–Uryasev
|
||||||
|
model in weights to whole shares with rebalancing and costs; sampling error of scenario sets, quasi-Monte Carlo,
|
||||||
|
moment matching, importance sampling, row generation, the exact Gaussian/mean–variance case and reproducibility;
|
||||||
|
source `docs/optimization_model.tex`),
|
||||||
|
[docs/measurement_and_models.pdf](docs/measurement_and_models.pdf) (risk and performance measures, rolling/EWM
|
||||||
|
estimators and shrinkage, GBM and empirical-innovation scenarios, Black–Litterman with Idzorek confidence;
|
||||||
|
source `docs/measurement_and_models.tex`),
|
||||||
|
[docs/SPARK_APPTAINER.md](docs/SPARK_APPTAINER.md) (**students: running it on your own DGX Spark** with
|
||||||
|
Apptainer, no root), [docs/SPARK_DEPLOYMENT.md](docs/SPARK_DEPLOYMENT.md) (the Docker setup on a Spark with root).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick start on your DGX Spark (students)
|
||||||
|
|
||||||
|
The service and cuOpt run as two Apptainer containers on your Spark and talk over `127.0.0.1`.
|
||||||
|
No root rights, no Docker, nothing exposed to the network:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management.git
|
||||||
|
cd Portfolio_Risk_Management
|
||||||
|
deploy/apptainer/qc.sh install # once: cuOpt image, quantcore image, self-test
|
||||||
|
deploy/apptainer/qc.sh data-import quantcore_data.tar.gz # prepared data set (or: qc.sh setup, ~15 min)
|
||||||
|
deploy/apptainer/qc.sh start # -> http://127.0.0.1:5001/ui/ and /docs
|
||||||
|
deploy/apptainer/qc.sh api selftest # every endpoint, pass/fail
|
||||||
|
```
|
||||||
|
|
||||||
|
Your platform then calls the API at `http://127.0.0.1:5001`. The full guide, with settings,
|
||||||
|
several users on one Spark and troubleshooting, is [docs/SPARK_APPTAINER.md](docs/SPARK_APPTAINER.md).
|
||||||
|
|
||||||
|
## Quick start (local)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e ".[dev]" # Python 3.11+
|
||||||
|
cp .env.example .env # optional: data dir, cuOpt URL, ...
|
||||||
|
|
||||||
|
quantcore setup # once: universe + full history (~15 min, ~1100 Yahoo tickers)
|
||||||
|
quantcore status # what is stored, cuOpt health
|
||||||
|
quantcore update # later: newest prices / FX (incremental)
|
||||||
|
quantcore snapshot --name demo # freeze the data for a reproducible demo
|
||||||
|
|
||||||
|
quantcore risk NESN.SW AAPL ASML.AS --horizon week --window 200 --lam 0.94
|
||||||
|
quantcore performance NESN.SW AAPL --freq weekly
|
||||||
|
quantcore optimize --cash 1000000 --cvar-limit 40000 --confidence 0.95 --horizon week --max-assets 100
|
||||||
|
quantcore serve # HTTP API on :5001 -> http://localhost:5001/docs
|
||||||
|
# and the test bench -> http://localhost:5001/ui/
|
||||||
|
```
|
||||||
|
|
||||||
|
Against a *running* service -- from any machine, without the data directory -- use the API client
|
||||||
|
instead (`quantcore-api`, or `quantcore.client.QuantcoreClient` from Python):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export QUANTCORE_API_URL=http://127.0.0.1:5001 # or the address of the machine that runs it
|
||||||
|
quantcore-api health
|
||||||
|
quantcore-api selftest # call every endpoint, pass/fail table, exit code
|
||||||
|
quantcore-api endpoints # the API surface, from its own OpenAPI schema
|
||||||
|
quantcore-api risk NESN.SW AAPL --horizon week --drift zero --space value
|
||||||
|
quantcore-api optimize --cash 1000000 --cvar-limit 40000 --horizon week --risk-drift zero
|
||||||
|
```
|
||||||
|
|
||||||
|
`quantcore setup` fetches the candidate lists from the web (Wikipedia S&P 500, iShares STOXX 600
|
||||||
|
holdings); `--offline` uses the snapshots shipped in `quantcore/data/static/`. All data lands in
|
||||||
|
`QUANTCORE_DATA_DIR` (default `./data`): `universe/universe.csv`, `prices/adj_close.parquet`,
|
||||||
|
`fx/fx_chf.parquet`, `manifest.json`, `snapshots/`.
|
||||||
|
|
||||||
|
## Using the services from Python
|
||||||
|
|
||||||
|
```python
|
||||||
|
from quantcore import get_settings
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.measurement.risk import RiskService
|
||||||
|
from quantcore.measurement.performance import PerformanceService
|
||||||
|
from quantcore.optimization.service import OptimizationService, OptimizationRequest
|
||||||
|
|
||||||
|
settings = get_settings()
|
||||||
|
md = MarketDataService(settings) # or MarketDataService(settings, snapshot="demo")
|
||||||
|
|
||||||
|
risk = RiskService(md)
|
||||||
|
risk.asset_risk(["NESN.SW", "AAPL"], horizon="week", window=200, lam=0.94) # per-asset table
|
||||||
|
risk.correlation(["NESN.SW", "AAPL", "ASML.AS"], method="ewm", lam=0.99)
|
||||||
|
risk.portfolio_risk({"NESN.SW": 500, "AAPL": 100}, cash=20_000, horizon="month", method="historical")
|
||||||
|
|
||||||
|
PerformanceService(md).asset_performance(["NESN.SW", "AAPL"], freq="weekly")
|
||||||
|
|
||||||
|
opt = OptimizationService(settings, md)
|
||||||
|
res = opt.optimize(OptimizationRequest(
|
||||||
|
mode="new", cash=1_000_000, max_assets=100, horizon="week", confidence=0.95, cvar_limit=40_000,
|
||||||
|
scenario_model="empirical", covariance={"method": "ewm", "lam": 0.99},
|
||||||
|
views=[{"symbol": "NESN.SW", "sentiment": 0.7, "confidence": 0.6, "source": "insight-1234"}],
|
||||||
|
))
|
||||||
|
res.positions_frame() # shares, value, weight, trade per stock
|
||||||
|
res.cvar, res.expected_pnl # CHF over the horizon
|
||||||
|
res.scenario_hash, res.reproducibility # same inputs -> same scenarios; 'exact' (HiGHS) | 'within_gap' (cuOpt)
|
||||||
|
res.scenario_key["id"] # varying the risk limit, constraints, costs or views keeps this id: one
|
||||||
|
# scenario set for a whole family of runs (and it is cached in the service)
|
||||||
|
|
||||||
|
# scenario options: 4096 moment-matched Sobol' scenarios by default; importance sampling for the tail
|
||||||
|
# switches on automatically at beta = 0.99; large sets are solved by tail-row generation
|
||||||
|
res99 = opt.optimize(OptimizationRequest(cash=1_000_000, confidence=0.99, cvar_limit=60_000, n_scenarios=16_384))
|
||||||
|
# the exact Gaussian (mean-variance-equivalent) solution: no scenarios, no sampling error
|
||||||
|
res_g = opt.optimize(OptimizationRequest(cash=1_000_000, cvar_limit=40_000, scenario_model="gaussian_exact"))
|
||||||
|
# n_scenarios defaults to 16 per candidate (4096-16384); the result warns if that is low for the
|
||||||
|
# candidate count or if the covariance estimate is ill-conditioned for it
|
||||||
|
|
||||||
|
# rebalancing: existing holdings, transaction costs, turnover limit
|
||||||
|
res2 = opt.optimize(OptimizationRequest(
|
||||||
|
mode="rebalance", cash=res.cash_end, holdings={p["symbol"]: p["shares"] for p in res.positions},
|
||||||
|
cvar_limit=30_000, costs={"proportional_buy": 0.001, "proportional_sell": 0.001, "fixed_per_trade": 10},
|
||||||
|
constraints={"max_weight": 0.10, "max_sector_weight": 0.25, "max_turnover": 0.3},
|
||||||
|
))
|
||||||
|
```
|
||||||
|
|
||||||
|
## HTTP API (the platform's contract)
|
||||||
|
|
||||||
|
| Method | Path | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| GET | `/health` | store status, cuOpt health |
|
||||||
|
| GET | `/universe?region=§or=&country=` · `/universe/{symbol}` | universe with metadata |
|
||||||
|
| GET | `/prices?symbols=A,B&freq=weekly¤cy=base` · `/returns?...` | prices / log-returns |
|
||||||
|
| GET | `/market/index?freq=weekly` · `/market/weights` | cap-weighted market index, weights |
|
||||||
|
| POST | `/risk/assets` (`space: log｜value`, `drift: estimated｜zero`) · `/risk/series` · `/risk/covariance` · `/risk/correlation` · `/risk/portfolio` | risk measurement |
|
||||||
|
| POST | `/performance/assets` · `/performance/rolling` · `/performance/portfolio` | performance measurement |
|
||||||
|
| GET | `/data/symbols?region=§or=&country=&search=` | one table of the universe (metadata, market cap, price) plus the filter facets |
|
||||||
|
| POST | `/model/expected-returns` · `/model/covariance` · `/model/scenarios` | Black–Litterman posterior, the estimator the models use, scenario set (+ the P&L distribution of a portfolio) |
|
||||||
|
| POST | `/optimize` | mean–CVaR MILP (new / rebalance; a rebalancing also reports the starting portfolio under `before`) |
|
||||||
|
| POST | `/jobs/setup` · `/jobs/update?full=` · GET `/jobs/{id}` | background setup / data update |
|
||||||
|
| POST/GET | `/snapshots` | freeze / list data snapshots |
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://127.0.0.1:5001/optimize -H 'Content-Type: application/json' -d '{
|
||||||
|
"mode": "new", "cash": 1000000, "max_assets": 100, "horizon": "week",
|
||||||
|
"confidence": 0.95, "cvar_limit": 40000, "scenario_model": "empirical",
|
||||||
|
"views": [{"symbol": "NESN.SW", "sentiment": 0.7, "confidence": 0.6}]
|
||||||
|
}' | python -m json.tool
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test bench (browser UI)
|
||||||
|
|
||||||
|
`http://127.0.0.1:5001/ui/` serves a single-page test bench — the
|
||||||
|
one place to drive every service by hand and judge what it produces. It is a pure client of the HTTP
|
||||||
|
API above: no numbers are computed in the browser.
|
||||||
|
|
||||||
|
Because it is a client and not a part of the API, it can be left out of a deployment:
|
||||||
|
`QUANTCORE_ENABLE_UI=false` (read once at startup) serves the JSON API and `/docs` only, `/ui/`
|
||||||
|
answers 404 and `/` no longer redirects. The same image therefore also suits an installation whose
|
||||||
|
users build their own front end against these endpoints.
|
||||||
|
|
||||||
|
The left panel holds the state every tab shares — the candidate stocks, the portfolio (new or
|
||||||
|
rebalancing, cash, holdings) and the models (frequency, horizon, stock-price model, scenario count,
|
||||||
|
seed, sampling, tail tilt, covariance estimator, expected-return model). Five tabs use it:
|
||||||
|
|
||||||
|
| Tab | What it answers |
|
||||||
|
|---|---|
|
||||||
|
| **Stocks & data** | Which stocks exist (filter by region, sector, country, name) and what their history looks like: indexed prices, drawdowns, risk/return, market caps, and a per-stock table of volatility, VaR/CVaR (normal and historical), Sharpe, drawdown, skew and kurtosis. |
|
||||||
|
| **Risk & scenarios** | Per-stock scenario moments and quantiles in **price returns** (bounded below by −100 %) or, with one click, in the log-returns the model works in — a quarterly log-return of −2.0 is a price return of −86 %, and mixing the two is the easiest way to misread a simulation. The covariance the optimiser actually sees: correlation heat map, volatilities, eigenvalue spectrum, shrinkage and conditioning warnings. Then the scenario set: the horizon P&L distribution of a chosen portfolio (cap-weighted, equally weighted, your holdings or the last proposal) with VaR/CVaR markers, per-stock moments and quantiles, a single-stock distribution, the scenario hash, and a sampling-noise check across three seeds. |
|
||||||
|
| **Expected returns** | The market-implied (equilibrium) returns from reverse optimisation, the views you add (sentiment or explicit, with Idzorek confidence), and the posterior — side by side, with the change per stock and how each view was translated into `q` and `ω`. |
|
||||||
|
| **Constraints** | Weight, sector, cardinality, turnover, cost and investment-band limits, transaction costs, plus a feasibility check against the candidate set (are there enough names, do the sector caps leave room, is any stock unaffordable at one share?) and the sector composition against the caps. |
|
||||||
|
| **Optimisation** | Objective, confidence, CVaR limit, the risk convention (`risk_drift`), solver and time limit; the proposal in integer shares with trades and costs; for a rebalancing the **before and after** of the same portfolio on the same scenarios; the weights and sector exposure against the limits; the provenance (scenario key and hash, solver, MIP gap, row-generation passes, reproducibility); a run history and a risk-limit sweep that traces the risk/return trade-off. |
|
||||||
|
|
||||||
|
A third: **the CVaR limit can stop constraining anything over a long horizon**. CVaR₉₉ ≈ −μₕ + 2.67 σₕ,
|
||||||
|
so with a one-year horizon and an expected return of 40 % against a portfolio volatility of ~13 %, the whole
|
||||||
|
1 % tail is a gain, the limit is slack and mean-CVaR degenerates into "buy the highest forecast returns up to
|
||||||
|
the weight caps". The result then warns, and `risk_drift="zero"` measures the CVaR on the centred scenarios
|
||||||
|
(the risk figure is then independent of the return forecast) while the objective keeps it. The `historical`
|
||||||
|
expected-return model makes this extreme: on this data it gives up to 821 % p.a. to a single stock.
|
||||||
|
|
||||||
|
Two things to know while using it. The **time limit is the budget of the whole run**, not of each
|
||||||
|
solver call: row generation and the Gaussian outer approximation solve a sequence of MILPs, and each
|
||||||
|
one gets what is left of the budget (the default is 300 s; an empty field means no limit, and HiGHS
|
||||||
|
will then run until it can prove optimality -- hours on a large candidate set). And the service
|
||||||
|
answers **one request at a time** while a solve runs: the MILP holds the interpreter, so the other
|
||||||
|
tabs wait until it finishes. The progress panel says both, and a run can be cancelled.
|
||||||
|
|
||||||
|
Typical checks it is built for: put 1 m or 50 m CHF into a new portfolio and see how the composition
|
||||||
|
changes with wealth (integer shares start to matter); restrict the universe to European stocks and
|
||||||
|
compare; add a view on one stock and watch the posterior and the proposal move; switch the
|
||||||
|
stock-price model from GBM to empirical innovations (or to the exact Gaussian mode, which has no
|
||||||
|
sampling error at all) and see the risk figures change; run the same request twice and confirm that
|
||||||
|
the scenario hash and the shares are identical.
|
||||||
|
|
||||||
|
## On the DGX Spark
|
||||||
|
|
||||||
|
There are two setups. Both run the same code and give the same results.
|
||||||
|
|
||||||
|
* **Apptainer, no root** (the students' Sparks): `deploy/apptainer/qc.sh` runs cuOpt and the API as
|
||||||
|
two Apptainer instances of the user, bound to `127.0.0.1`. See
|
||||||
|
[docs/SPARK_APPTAINER.md](docs/SPARK_APPTAINER.md).
|
||||||
|
* **Docker Compose, with root** (the lecturer's Spark): `deploy/deploy_to_spark.sh --with-data`
|
||||||
|
adds a `quantcore` service next to the `cuopt` service of an existing compose file and reaches
|
||||||
|
cuOpt at `http://cuopt:5000`. See [docs/SPARK_DEPLOYMENT.md](docs/SPARK_DEPLOYMENT.md).
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest # 127 tests: reference values for VaR/CVaR, Idzorek closed form vs. numeric, brute-force MILP optimality,
|
||||||
|
# exact Gaussian mode vs. brute force, row generation vs. full MILP, sampling/moment matching/tilt,
|
||||||
|
# weighted quantiles and scenario distributions, API (incl. the before/after of a rebalancing),
|
||||||
|
# the HTTP client and every `quantcore-api` command against the app in-process
|
||||||
|
```
|
||||||
|
|
||||||
|
The cuOpt round-trip test runs only when the server is reachable.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
quantcore/
|
||||||
|
config.py settings (env QUANTCORE_*, .env)
|
||||||
|
universe/ candidate sources, Yahoo symbol mapping, SetupService
|
||||||
|
data/ YahooClient (timeouts/retry/breaker), FX, DataStore, DataDownloadService, MarketDataService, static/
|
||||||
|
measurement/ returns, rolling & EWM estimators, risk (VaR/CVaR), performance
|
||||||
|
models/ CovarianceModel + matrix roots, GBM & empirical-innovation simulators, Black-Litterman/Idzorek
|
||||||
|
optimization/ MILP builder (problem.py), cuOpt/HiGHS solvers, OptimizationService
|
||||||
|
api/ FastAPI app + pydantic schemas
|
||||||
|
api/ui/ the browser test bench (static HTML/CSS/ES modules, no build step, no CDN);
|
||||||
|
QUANTCORE_ENABLE_UI=false serves the JSON API and /docs only
|
||||||
|
client.py QuantcoreClient -- the API as a Python object (needs only a URL)
|
||||||
|
cli.py `quantcore` command (in-process services, needs the data directory)
|
||||||
|
apicli.py `quantcore-api` command (HTTP client of a running service, incl. `selftest`)
|
||||||
|
deploy/apptainer/ qc.sh (install/start/stop/cli on a Spark without root), quantcore.def, pinned constraints.txt
|
||||||
|
deploy/ Dockerfile, compose service blocks (Spark + a bench-free one), deploy_to_spark.sh
|
||||||
|
docs/ USAGE.md, ARCHITECTURE.md, METHODOLOGY.md, SPARK_APPTAINER.md, SPARK_DEPLOYMENT.md
|
||||||
|
tests/ pytest suite (synthetic data store fixture)
|
||||||
|
```
|
||||||
|
|||||||
24
deploy/Dockerfile
Normal file
24
deploy/Dockerfile
Normal file
@ -0,0 +1,24 @@
|
|||||||
|
# quantcore -- portfolio quant service for the DGX Spark (aarch64) and any x86-64 host.
|
||||||
|
# Build context is the repository root: docker build -f deploy/Dockerfile -t local/quantcore:0.1 .
|
||||||
|
FROM python:3.12-slim
|
||||||
|
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PIP_NO_CACHE_DIR=1 \
|
||||||
|
QUANTCORE_DATA_DIR=/data QUANTCORE_CUOPT_URL=http://cuopt:5000 \
|
||||||
|
OPENBLAS_NUM_THREADS=4 OMP_NUM_THREADS=4 MKL_NUM_THREADS=4
|
||||||
|
# BLAS threading: the innovation filter runs thousands of small (N x N) eigendecompositions;
|
||||||
|
# OpenBLAS with all 20 cores of the Spark is ~50x slower on those than with 1-4 threads.
|
||||||
|
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends curl tini \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
COPY pyproject.toml README.md ./
|
||||||
|
COPY quantcore ./quantcore
|
||||||
|
RUN pip install .
|
||||||
|
|
||||||
|
VOLUME ["/data"]
|
||||||
|
EXPOSE 5001
|
||||||
|
HEALTHCHECK --interval=60s --timeout=10s --retries=2 CMD curl -sf http://localhost:5001/health >/dev/null || exit 1
|
||||||
|
|
||||||
|
ENTRYPOINT ["/usr/bin/tini", "--"]
|
||||||
|
CMD ["uvicorn", "quantcore.api.app:app", "--host", "0.0.0.0", "--port", "5001", "--workers", "1"]
|
||||||
48
deploy/apptainer/constraints.txt
Normal file
48
deploy/apptainer/constraints.txt
Normal file
@ -0,0 +1,48 @@
|
|||||||
|
# Library versions of the quantcore image, taken from the container that runs on the lecturer's Spark
|
||||||
|
# (Python 3.12.14, aarch64, 2026-10-07). deploy/apptainer/quantcore.def installs with `pip install -c`,
|
||||||
|
# so a student's build in December gets the same SciPy/HiGHS, pandas and numpy as the verified one.
|
||||||
|
# To move to newer libraries: delete the pins, build, run the tests, and freeze again:
|
||||||
|
# apptainer exec quantcore.sif pip freeze --exclude quantcore
|
||||||
|
annotated-doc==0.0.5
|
||||||
|
annotated-types==0.8.0
|
||||||
|
anyio==4.15.1
|
||||||
|
beautifulsoup4==4.15.0
|
||||||
|
certifi==2026.7.22
|
||||||
|
cffi==2.1.1
|
||||||
|
charset-normalizer==3.5.1
|
||||||
|
click==8.5.0
|
||||||
|
curl_cffi==0.16.3
|
||||||
|
fastapi==0.141.1
|
||||||
|
h11==0.16.0
|
||||||
|
httptools==0.8.0
|
||||||
|
idna==3.20
|
||||||
|
lxml==6.1.3
|
||||||
|
msgpack==1.2.2
|
||||||
|
multitasking==0.0.13
|
||||||
|
numpy==2.5.3
|
||||||
|
pandas==3.0.6
|
||||||
|
peewee==4.5.1
|
||||||
|
platformdirs==4.11.12
|
||||||
|
protobuf==7.36.2
|
||||||
|
pyarrow==25.0.1
|
||||||
|
pycparser==3.0
|
||||||
|
pydantic==2.13.5
|
||||||
|
pydantic-settings==2.15.0
|
||||||
|
pydantic_core==2.46.5
|
||||||
|
python-dateutil==2.9.0.post0
|
||||||
|
python-dotenv==1.2.3
|
||||||
|
pytz==2026.3.post1
|
||||||
|
PyYAML==6.0.3
|
||||||
|
requests==2.34.2
|
||||||
|
scipy==1.18.1
|
||||||
|
six==1.17.0
|
||||||
|
soupsieve==2.9.2
|
||||||
|
starlette==1.7.0
|
||||||
|
typing-inspection==0.4.4
|
||||||
|
typing_extensions==4.16.0
|
||||||
|
urllib3==2.8.0
|
||||||
|
uvicorn==0.53.0
|
||||||
|
uvloop==0.22.1
|
||||||
|
watchfiles==1.3.0
|
||||||
|
websockets==17.1
|
||||||
|
yfinance==1.7.0
|
||||||
576
deploy/apptainer/qc.sh
Executable file
576
deploy/apptainer/qc.sh
Executable file
@ -0,0 +1,576 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# qc.sh -- quantcore standalone on a DGX Spark with Apptainer: no root, no Docker, no Tailscale.
|
||||||
|
#
|
||||||
|
# cuOpt and quantcore each run in their own Apptainer container on this machine and talk over
|
||||||
|
# 127.0.0.1. The API and the browser test bench listen on 127.0.0.1 too, so they are used from this
|
||||||
|
# Spark (its desktop browser, a terminal, a notebook) and are not reachable from the network.
|
||||||
|
#
|
||||||
|
# qc.sh install once: config, cuOpt image, quantcore image, data directory
|
||||||
|
# qc.sh data-import FILE copy a prepared data set (.tar.gz or directory) ...
|
||||||
|
# qc.sh setup ... or build it from Yahoo Finance (~15 min)
|
||||||
|
# qc.sh start cuOpt + API -> http://127.0.0.1:5001/ui/ and /docs
|
||||||
|
# qc.sh status | logs [api|cuopt] [-f] | stop | restart
|
||||||
|
# qc.sh cli <command> ... the quantcore CLI in the container, no API needed
|
||||||
|
# (e.g. qc.sh cli optimize --cash 1e6 --cvar-limit 40000)
|
||||||
|
# qc.sh api <command> ... quantcore-api against the local API (e.g. qc.sh api selftest)
|
||||||
|
# qc.sh help all commands
|
||||||
|
#
|
||||||
|
# Settings: $QC_HOME/quantcore.env (QC_HOME defaults to ~/quantcore); see quantcore.env.example.
|
||||||
|
# Guide: docs/SPARK_APPTAINER.md.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
REPO_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||||
|
QC_HOME="${QC_HOME:-$HOME/quantcore}"
|
||||||
|
CONFIG="$QC_HOME/quantcore.env"
|
||||||
|
APPTAINER="${APPTAINER:-apptainer}"
|
||||||
|
API_INSTANCE="quantcore-api"
|
||||||
|
CUOPT_INSTANCE="quantcore-cuopt"
|
||||||
|
ME="$(id -un)"
|
||||||
|
|
||||||
|
# --- settings: defaults < config file < environment ------------------------------------------
|
||||||
|
_env_overrides="$(declare -p $(compgen -v | grep -E '^(QC_|QUANTCORE_)' || true) 2>/dev/null || true)"
|
||||||
|
QC_DATA="$QC_HOME/data"
|
||||||
|
QC_IMAGES="$QC_HOME/images"
|
||||||
|
QC_API_PORT=5001
|
||||||
|
QC_CUOPT_PORT=5000
|
||||||
|
QC_CUOPT_START=auto
|
||||||
|
QC_CUOPT_IMAGE="docker://nvidia/cuopt:26.8.0-cuda13.3-py3.14"
|
||||||
|
QC_IMAGE=build
|
||||||
|
QC_PYTHON_IMAGE="docker://python:3.12-slim" # base of the no-build fallback (same as quantcore.def)
|
||||||
|
if [[ -f "$CONFIG" ]]; then
|
||||||
|
set -a
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
source "$CONFIG"
|
||||||
|
set +a
|
||||||
|
fi
|
||||||
|
[[ -n "$_env_overrides" ]] && eval "$_env_overrides"
|
||||||
|
unset _env_overrides
|
||||||
|
QC_API_URL="http://127.0.0.1:$QC_API_PORT"
|
||||||
|
QC_VENV="$QC_HOME/venv"
|
||||||
|
PY_SIF="$QC_IMAGES/python.sif"
|
||||||
|
CUOPT_URL="${QUANTCORE_CUOPT_URL:-http://127.0.0.1:$QC_CUOPT_PORT}"
|
||||||
|
CUOPT_EXTERNAL=0 # 1: QUANTCORE_CUOPT_URL names a cuOpt somewhere else -- never start our own
|
||||||
|
[[ -n "${QUANTCORE_CUOPT_URL:-}" ]] && CUOPT_EXTERNAL=1
|
||||||
|
|
||||||
|
# --- helpers ----------------------------------------------------------------------------------
|
||||||
|
say() { printf '%s\n' "$*"; }
|
||||||
|
step() { printf '\n>> %s\n' "$*"; }
|
||||||
|
warn() { printf 'warning: %s\n' "$*" >&2; }
|
||||||
|
die() { printf 'error: %s\n' "$*" >&2; exit 1; }
|
||||||
|
|
||||||
|
quantcore_sif() { if [[ "$QC_IMAGE" == *.sif && -f "$QC_IMAGE" ]]; then say "$QC_IMAGE"; else say "$QC_IMAGES/quantcore.sif"; fi; }
|
||||||
|
|
||||||
|
# How quantcore runs: "sif" = the quantcore image (built or prebuilt); "venv" = the no-build fallback,
|
||||||
|
# a plain Python image plus a virtualenv in $QC_HOME/venv (for a Spark that allows no image builds).
|
||||||
|
qc_mode() {
|
||||||
|
if [[ "$QC_IMAGE" == "venv" ]]; then say venv; return; fi
|
||||||
|
if [[ -f "$(quantcore_sif)" ]]; then say sif; return; fi
|
||||||
|
if [[ -x "$QC_VENV/bin/quantcore" && -f "$PY_SIF" ]]; then say venv; return; fi
|
||||||
|
say none
|
||||||
|
}
|
||||||
|
qc_image() { if [[ "$(qc_mode)" == venv ]]; then say "$PY_SIF"; else quantcore_sif; fi; }
|
||||||
|
qc_bin() { if [[ "$(qc_mode)" == venv ]]; then say "$QC_VENV/bin/$1"; else say "$1"; fi; }
|
||||||
|
cuopt_sif() { if [[ -f "$QC_CUOPT_IMAGE" ]]; then say "$QC_CUOPT_IMAGE"; else say "$QC_IMAGES/cuopt.sif"; fi; }
|
||||||
|
|
||||||
|
need_apptainer() {
|
||||||
|
command -v "$APPTAINER" >/dev/null 2>&1 \
|
||||||
|
|| die "'$APPTAINER' not found. This Spark needs Apptainer (ask the administrator), or set APPTAINER=singularity."
|
||||||
|
}
|
||||||
|
|
||||||
|
need_image() { # path what
|
||||||
|
[[ -f "$1" ]] || die "$2 image missing ($1) -- run: $0 install"
|
||||||
|
}
|
||||||
|
|
||||||
|
# HTTP GET to stdout; exit status 0 only for a 2xx answer. curl, else python3 (both on DGX OS).
|
||||||
|
http_get() {
|
||||||
|
local url="$1" timeout="${2:-3}"
|
||||||
|
if command -v curl >/dev/null 2>&1; then
|
||||||
|
curl -sf --max-time "$timeout" "$url"
|
||||||
|
elif command -v python3 >/dev/null 2>&1; then
|
||||||
|
python3 -c 'import sys, urllib.request as u; sys.stdout.write(u.urlopen(sys.argv[1], timeout=float(sys.argv[2])).read().decode())' \
|
||||||
|
"$url" "$timeout" 2>/dev/null
|
||||||
|
else
|
||||||
|
return 2
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
port_busy() { (exec 3<>"/dev/tcp/127.0.0.1/$1") 2>/dev/null; }
|
||||||
|
|
||||||
|
free_port() { # first free port from $1 upwards
|
||||||
|
local p="$1"
|
||||||
|
while port_busy "$p"; do p=$((p + 1)); done
|
||||||
|
say "$p"
|
||||||
|
}
|
||||||
|
|
||||||
|
instance_running() {
|
||||||
|
"$APPTAINER" instance list 2>/dev/null | awk 'NR > 1 {print $1}' | grep -qx "$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
instance_log() { # name out|err -> path of the instance log (Apptainer keeps them per host and user)
|
||||||
|
local f
|
||||||
|
for f in "$HOME/.apptainer/instances/logs/"*"/$ME/$1.$2"; do
|
||||||
|
[[ -f "$f" ]] && { say "$f"; return 0; }
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
show_log_tail() { # name
|
||||||
|
local f
|
||||||
|
for kind in err out; do
|
||||||
|
if f="$(instance_log "$1" "$kind")"; then
|
||||||
|
say "--- last lines of $f"
|
||||||
|
tail -n 25 "$f" >&2 || true
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
wait_for() { # url seconds label
|
||||||
|
local url="$1" limit="$2" label="$3" waited=0
|
||||||
|
printf ' waiting for %s ' "$label"
|
||||||
|
until http_get "$url" 2 >/dev/null; do
|
||||||
|
if (( waited >= limit )); then printf ' timed out after %ss\n' "$limit"; return 1; fi
|
||||||
|
printf '.'; sleep 2; waited=$((waited + 2))
|
||||||
|
done
|
||||||
|
printf ' up (%ss)\n' "$waited"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Hand the settings to a container through APPTAINERENV_* (unlike --env these survive commas,
|
||||||
|
# e.g. in no_proxy). The containers start with --cleanenv, so nothing else from the host leaks in:
|
||||||
|
# no PYTHONPATH, no activated virtualenv or conda, no LD_LIBRARY_PATH.
|
||||||
|
export_quantcore_env() {
|
||||||
|
local v
|
||||||
|
for v in $(compgen -v | grep '^QUANTCORE_' || true); do
|
||||||
|
export "APPTAINERENV_$v=${!v}"
|
||||||
|
done
|
||||||
|
export APPTAINERENV_QUANTCORE_DATA_DIR=/data
|
||||||
|
export APPTAINERENV_QUANTCORE_CUOPT_URL="$CUOPT_URL"
|
||||||
|
export APPTAINERENV_QUANTCORE_API_HOST=127.0.0.1
|
||||||
|
export APPTAINERENV_QUANTCORE_API_PORT="$QC_API_PORT"
|
||||||
|
export APPTAINERENV_QUANTCORE_API_URL="$QC_API_URL"
|
||||||
|
# what quantcore.def's %environment sets, for the venv fallback (an explicit value wins)
|
||||||
|
export APPTAINERENV_PYTHONNOUSERSITE=1 APPTAINERENV_PYTHONUNBUFFERED=1 APPTAINERENV_PYTHONDONTWRITEBYTECODE=1
|
||||||
|
export APPTAINERENV_OPENBLAS_NUM_THREADS="${OPENBLAS_NUM_THREADS:-4}" APPTAINERENV_OMP_NUM_THREADS="${OMP_NUM_THREADS:-4}"
|
||||||
|
export APPTAINERENV_MKL_NUM_THREADS="${MKL_NUM_THREADS:-4}"
|
||||||
|
for v in http_proxy https_proxy no_proxy HTTP_PROXY HTTPS_PROXY NO_PROXY TZ; do
|
||||||
|
[[ -n "${!v:-}" ]] && export "APPTAINERENV_$v=${!v}"
|
||||||
|
done
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# Run a command in the quantcore container (the CLI needs no running API).
|
||||||
|
qc_binds() { # bind options for the quantcore container
|
||||||
|
local b=(--bind "$QC_DATA:/data")
|
||||||
|
[[ "$(qc_mode)" == venv ]] && b+=(--bind "$QC_VENV")
|
||||||
|
printf '%s\n' "${b[@]}"
|
||||||
|
}
|
||||||
|
|
||||||
|
run_qc() { # command [args]: quantcore | quantcore-api | python ...
|
||||||
|
need_apptainer
|
||||||
|
[[ "$(qc_mode)" != none ]] || die "no quantcore image yet -- run: $0 install"
|
||||||
|
mkdir -p "$QC_DATA"
|
||||||
|
export_quantcore_env
|
||||||
|
local cmd="$1"; shift
|
||||||
|
local binds; mapfile -t binds < <(qc_binds)
|
||||||
|
exec "$APPTAINER" exec --cleanenv "${binds[@]}" "$(qc_image)" "$(qc_bin "$cmd")" "$@"
|
||||||
|
}
|
||||||
|
|
||||||
|
data_is_empty() { [[ -z "$(ls -A "$QC_DATA" 2>/dev/null | grep -v -x -e jobs -e '.*\.log')" ]]; }
|
||||||
|
|
||||||
|
# --- install ----------------------------------------------------------------------------------
|
||||||
|
cmd_install() {
|
||||||
|
need_apptainer
|
||||||
|
local force=0; [[ "${1:-}" == "--force" ]] && force=1
|
||||||
|
step "Apptainer"
|
||||||
|
say " $("$APPTAINER" --version)"
|
||||||
|
"$APPTAINER" instance run --help >/dev/null 2>&1 \
|
||||||
|
|| die "this Apptainer has no 'instance run' (needs Apptainer >= 1.1); ask the administrator to update it"
|
||||||
|
|
||||||
|
step "Directories and settings ($QC_HOME)"
|
||||||
|
mkdir -p "$QC_HOME" "$QC_DATA" "$QC_IMAGES"
|
||||||
|
if [[ -f "$CONFIG" ]]; then
|
||||||
|
say " keeping $CONFIG"
|
||||||
|
else
|
||||||
|
cp "$SCRIPT_DIR/quantcore.env.example" "$CONFIG"
|
||||||
|
say " wrote $CONFIG"
|
||||||
|
fi
|
||||||
|
|
||||||
|
step "GPU"
|
||||||
|
if command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi -L >/dev/null 2>&1; then
|
||||||
|
say " $(nvidia-smi -L | head -n 1)"
|
||||||
|
else
|
||||||
|
warn "no GPU visible (nvidia-smi). cuOpt will not start; optimisations will use HiGHS on the CPU."
|
||||||
|
fi
|
||||||
|
local free_gb; free_gb=$(( $(df -Pk "$QC_IMAGES" | awk 'NR == 2 {print $4}') / 1024 / 1024 ))
|
||||||
|
say " free disk under $QC_IMAGES: ${free_gb} GB (the cuOpt pull needs ~12 GB while it runs, 3.3 GB after)"
|
||||||
|
|
||||||
|
if [[ "$QC_CUOPT_START" != "no" && "$CUOPT_EXTERNAL" == 0 ]]; then
|
||||||
|
cmd_pull_cuopt $([[ $force == 1 ]] && echo --force)
|
||||||
|
fi
|
||||||
|
step "quantcore image"
|
||||||
|
get_quantcore_image "$force"
|
||||||
|
step "Self-test of the quantcore container ($(qc_mode) mode)"
|
||||||
|
qc_selftest
|
||||||
|
|
||||||
|
step "Data ($QC_DATA)"
|
||||||
|
if data_is_empty; then
|
||||||
|
say " empty. Either copy a prepared data set: $0 data-import quantcore_data.tar.gz"
|
||||||
|
say " or build it from Yahoo Finance (~15 min): $0 setup"
|
||||||
|
else
|
||||||
|
say " present ($(du -sh "$QC_DATA" | cut -f1))"
|
||||||
|
fi
|
||||||
|
step "Done. Next: $0 start"
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_pull_cuopt() {
|
||||||
|
need_apptainer
|
||||||
|
local dst="$QC_IMAGES/cuopt.sif"
|
||||||
|
if [[ -f "$QC_CUOPT_IMAGE" ]]; then
|
||||||
|
step "cuOpt image: $QC_CUOPT_IMAGE (provided on this machine, not pulled)"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
if [[ -f "$dst" && "${1:-}" != "--force" ]]; then
|
||||||
|
step "cuOpt image present: $dst"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
step "Pulling cuOpt: $QC_CUOPT_IMAGE -> $dst (~3.3 GB, several minutes)"
|
||||||
|
mkdir -p "$QC_IMAGES"
|
||||||
|
rm -f "$dst.part"
|
||||||
|
"$APPTAINER" pull "$dst.part" "$QC_CUOPT_IMAGE"
|
||||||
|
mv -f "$dst.part" "$dst"
|
||||||
|
}
|
||||||
|
|
||||||
|
qc_selftest() {
|
||||||
|
( export_quantcore_env
|
||||||
|
local binds; mapfile -t binds < <(qc_binds)
|
||||||
|
"$APPTAINER" exec --cleanenv "${binds[@]}" "$(qc_image)" "$(qc_bin python)" -c '
|
||||||
|
import importlib.resources as r, scipy, quantcore, quantcore.api.app as app
|
||||||
|
from scipy.optimize import milp
|
||||||
|
assert (r.files("quantcore.api") / "ui" / "index.html").is_file(), "test bench missing"
|
||||||
|
print(" quantcore", quantcore.__version__, "| scipy", scipy.__version__, "| UI mounted:", app.UI_ENABLED)' )
|
||||||
|
}
|
||||||
|
|
||||||
|
get_quantcore_image() { # force(0|1)
|
||||||
|
local dst="$QC_IMAGES/quantcore.sif"
|
||||||
|
case "$QC_IMAGE" in
|
||||||
|
build)
|
||||||
|
if [[ -f "$dst" && "$1" != 1 ]]; then say " present: $dst (rebuild: $0 build)"; return 0; fi
|
||||||
|
if [[ -x "$QC_VENV/bin/quantcore" && -f "$PY_SIF" && "$1" != 1 ]]; then
|
||||||
|
say " present: virtualenv $QC_VENV (no-build fallback; reinstall: $0 build)"; return 0
|
||||||
|
fi
|
||||||
|
build_sif || {
|
||||||
|
say ""
|
||||||
|
say " The image build is not possible on this Spark without root. Falling back to the no-build"
|
||||||
|
say " setup: a plain Python image plus a virtualenv in $QC_VENV (same libraries, same results)."
|
||||||
|
build_venv
|
||||||
|
} ;;
|
||||||
|
venv)
|
||||||
|
if [[ -x "$QC_VENV/bin/quantcore" && -f "$PY_SIF" && "$1" != 1 ]]; then say " present: $QC_VENV"; else build_venv; fi ;;
|
||||||
|
docker://*|oras://*|library://*|http://*|https://*)
|
||||||
|
if [[ -f "$dst" && "$1" != 1 ]]; then say " present: $dst"; else
|
||||||
|
say " pulling $QC_IMAGE -> $dst"
|
||||||
|
rm -f "$dst.part"; "$APPTAINER" pull "$dst.part" "$QC_IMAGE"; mv -f "$dst.part" "$dst"
|
||||||
|
fi ;;
|
||||||
|
*)
|
||||||
|
[[ -f "$QC_IMAGE" ]] || die "QC_IMAGE=$QC_IMAGE is neither 'build', a URL, nor an existing .sif file"
|
||||||
|
say " using $QC_IMAGE" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
build_sif() { # build quantcore.sif from the definition; non-zero when this Apptainer cannot build
|
||||||
|
local dst="$QC_IMAGES/quantcore.sif"
|
||||||
|
mkdir -p "$QC_IMAGES"
|
||||||
|
say " building $dst from $REPO_DIR (a few minutes)"
|
||||||
|
rm -f "$dst.part"
|
||||||
|
# %files paths in the definition are relative to the repository root. Without root Apptainer
|
||||||
|
# builds through --fakeroot (user namespaces); a Spark may not allow that.
|
||||||
|
if ! (cd "$REPO_DIR" && "$APPTAINER" build "$dst.part" deploy/apptainer/quantcore.def); then
|
||||||
|
rm -f "$dst.part"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
mv -f "$dst.part" "$dst"
|
||||||
|
say " built $dst ($(du -h "$dst" | cut -f1))"
|
||||||
|
}
|
||||||
|
|
||||||
|
build_venv() { # the no-build fallback: pull a plain Python image, install quantcore into a virtualenv
|
||||||
|
mkdir -p "$QC_IMAGES" "$QC_HOME"
|
||||||
|
if [[ ! -f "$PY_SIF" ]]; then
|
||||||
|
say " pulling $QC_PYTHON_IMAGE -> $PY_SIF"
|
||||||
|
rm -f "$PY_SIF.part"; "$APPTAINER" pull "$PY_SIF.part" "$QC_PYTHON_IMAGE"; mv -f "$PY_SIF.part" "$PY_SIF"
|
||||||
|
fi
|
||||||
|
say " installing quantcore into $QC_VENV (pinned versions from deploy/apptainer/constraints.txt)"
|
||||||
|
local work; work="$(mktemp -d "${TMPDIR:-/tmp}/quantcore-src.XXXXXX")"
|
||||||
|
# install from a copy: pip writes build/ and *.egg-info next to the sources
|
||||||
|
cp -r "$REPO_DIR/pyproject.toml" "$REPO_DIR/README.md" "$REPO_DIR/quantcore" "$work/"
|
||||||
|
"$APPTAINER" exec --cleanenv --bind "$QC_HOME" --bind "$work" "$PY_SIF" python -m venv --clear "$QC_VENV"
|
||||||
|
"$APPTAINER" exec --cleanenv --env PIP_DISABLE_PIP_VERSION_CHECK=1 --bind "$QC_HOME" --bind "$work" \
|
||||||
|
--bind "$REPO_DIR/deploy/apptainer/constraints.txt:/tmp/qc-constraints.txt" "$PY_SIF" \
|
||||||
|
"$QC_VENV/bin/pip" install --quiet --no-cache-dir -c /tmp/qc-constraints.txt "$work"
|
||||||
|
rm -rf "$work"
|
||||||
|
say " installed: $("$APPTAINER" exec --cleanenv --bind "$QC_VENV" "$PY_SIF" "$QC_VENV/bin/quantcore" --help >/dev/null && echo ok)"
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_build() { # rebuild after a code change, in the mode in use
|
||||||
|
need_apptainer
|
||||||
|
local mode; mode="$(qc_mode)"
|
||||||
|
step "Rebuilding quantcore ($([[ $mode == venv ]] && echo "virtualenv $QC_VENV" || echo "image $QC_IMAGES/quantcore.sif"))"
|
||||||
|
if [[ "$mode" == venv ]]; then
|
||||||
|
build_venv
|
||||||
|
elif [[ "$QC_IMAGE" == build ]]; then
|
||||||
|
build_sif || die "the image build failed (see above). Without root it needs Apptainer's --fakeroot. Use the
|
||||||
|
no-build fallback instead: set QC_IMAGE=venv in $CONFIG and run: $0 build"
|
||||||
|
else
|
||||||
|
die "QC_IMAGE=$QC_IMAGE is a prebuilt image; nothing to build"
|
||||||
|
fi
|
||||||
|
if instance_running "$API_INSTANCE"; then say " the running API still uses the old code: $0 restart"; fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- run --------------------------------------------------------------------------------------
|
||||||
|
start_cuopt() {
|
||||||
|
if [[ "$CUOPT_EXTERNAL" == 1 ]]; then
|
||||||
|
if http_get "$CUOPT_URL/cuopt/health" >/dev/null; then say "cuOpt: using $CUOPT_URL (QUANTCORE_CUOPT_URL)"
|
||||||
|
else warn "cuOpt at $CUOPT_URL (QUANTCORE_CUOPT_URL) does not answer -- optimisations will use HiGHS"; fi
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
if [[ "$QC_CUOPT_START" == "no" ]]; then
|
||||||
|
say "cuOpt: not started (QC_CUOPT_START=no) -- optimisations use HiGHS on the CPU"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
if http_get "$CUOPT_URL/cuopt/health" >/dev/null; then
|
||||||
|
if instance_running "$CUOPT_INSTANCE"; then
|
||||||
|
say "cuOpt: running ($CUOPT_INSTANCE, $CUOPT_URL)"
|
||||||
|
elif [[ "$QC_CUOPT_START" == "yes" ]]; then
|
||||||
|
die "QC_CUOPT_START=yes, but another cuOpt already serves port $QC_CUOPT_PORT. Use it (QC_CUOPT_START=auto)
|
||||||
|
or pick another port: QC_CUOPT_PORT=$(free_port $((QC_CUOPT_PORT + 100))) in $CONFIG"
|
||||||
|
else
|
||||||
|
say "cuOpt: using the server that already answers at $CUOPT_URL"
|
||||||
|
fi
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
if instance_running "$CUOPT_INSTANCE"; then # still starting, or hung
|
||||||
|
wait_for "$CUOPT_URL/cuopt/health" 180 "cuOpt ($CUOPT_INSTANCE)" && return 0
|
||||||
|
show_log_tail "$CUOPT_INSTANCE"
|
||||||
|
die "cuOpt instance runs but does not answer; try: $0 restart"
|
||||||
|
fi
|
||||||
|
if port_busy "$QC_CUOPT_PORT"; then
|
||||||
|
die "port $QC_CUOPT_PORT is taken by something that is not cuOpt.
|
||||||
|
Set QC_CUOPT_PORT=$(free_port $((QC_CUOPT_PORT + 100))) in $CONFIG and run: $0 start"
|
||||||
|
fi
|
||||||
|
local sif; sif="$(cuopt_sif)"
|
||||||
|
need_image "$sif" cuOpt
|
||||||
|
say "cuOpt: starting instance $CUOPT_INSTANCE on 127.0.0.1:$QC_CUOPT_PORT (GPU)"
|
||||||
|
# NVIDIA's entrypoint sets HOME=/opt/cuopt. That is writable under Docker but read-only in a SIF,
|
||||||
|
# and the solver process dies on its first job when CuPy cannot write its kernel cache there.
|
||||||
|
# So the server runs with HOME on a writable directory of ours (CuPy, Numba and CUDA caches).
|
||||||
|
local cuopt_home="$QC_HOME/cuopt-home"
|
||||||
|
mkdir -p "$cuopt_home"
|
||||||
|
( [[ -n "${CUDA_VISIBLE_DEVICES:-}" ]] && export APPTAINERENV_CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES"
|
||||||
|
cd /
|
||||||
|
# the image's entrypoint runs its arguments: the cuOpt REST server, on localhost only
|
||||||
|
"$APPTAINER" instance run --nv --cleanenv --bind "$cuopt_home" "$sif" "$CUOPT_INSTANCE" \
|
||||||
|
env HOME="$cuopt_home" python -m cuopt_server.cuopt_service \
|
||||||
|
-i 127.0.0.1 -p "$QC_CUOPT_PORT" -l "${QC_CUOPT_LOG_LEVEL:-info}" >/dev/null )
|
||||||
|
if ! wait_for "$CUOPT_URL/cuopt/health" 180 "cuOpt"; then
|
||||||
|
show_log_tail "$CUOPT_INSTANCE"
|
||||||
|
warn "cuOpt did not come up -- the API starts anyway and optimises with HiGHS"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
start_api() {
|
||||||
|
if instance_running "$API_INSTANCE"; then
|
||||||
|
if http_get "$QC_API_URL/health" >/dev/null; then say "API: running ($API_INSTANCE, $QC_API_URL)"; return 0; fi
|
||||||
|
wait_for "$QC_API_URL/health" 60 "API ($API_INSTANCE)" && return 0
|
||||||
|
show_log_tail "$API_INSTANCE"
|
||||||
|
die "the API instance runs but does not answer; try: $0 restart"
|
||||||
|
fi
|
||||||
|
if port_busy "$QC_API_PORT"; then
|
||||||
|
die "port $QC_API_PORT is taken (another user's quantcore?).
|
||||||
|
Set QC_API_PORT=$(free_port $((QC_API_PORT + 10))) in $CONFIG and run: $0 start"
|
||||||
|
fi
|
||||||
|
local mode; mode="$(qc_mode)"
|
||||||
|
[[ "$mode" != none ]] || die "no quantcore image yet -- run: $0 install"
|
||||||
|
mkdir -p "$QC_DATA"
|
||||||
|
say "API: starting instance $API_INSTANCE on $QC_API_URL ($mode mode)"
|
||||||
|
( export_quantcore_env
|
||||||
|
local binds; mapfile -t binds < <(qc_binds)
|
||||||
|
# start from /: pydantic-settings reads a .env from the working directory, and the container's
|
||||||
|
# / has none (instances take the caller's directory; they have no --cwd option)
|
||||||
|
cd /
|
||||||
|
if [[ "$mode" == venv ]]; then # the Python image's runscript executes its arguments
|
||||||
|
"$APPTAINER" instance run --cleanenv "${binds[@]}" "$PY_SIF" "$API_INSTANCE" "$QC_VENV/bin/quantcore" serve >/dev/null
|
||||||
|
else # quantcore.def's startscript: quantcore serve
|
||||||
|
"$APPTAINER" instance start --cleanenv "${binds[@]}" "$(quantcore_sif)" "$API_INSTANCE" >/dev/null
|
||||||
|
fi )
|
||||||
|
if ! wait_for "$QC_API_URL/health" 90 "API"; then
|
||||||
|
show_log_tail "$API_INSTANCE"
|
||||||
|
die "the API did not come up"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_start() {
|
||||||
|
need_apptainer
|
||||||
|
step "Starting quantcore (user $ME, settings $([[ -f "$CONFIG" ]] && echo "$CONFIG" || echo "defaults"))"
|
||||||
|
start_cuopt
|
||||||
|
start_api
|
||||||
|
data_is_empty && warn "no market data yet: $0 data-import <file> or $0 setup"
|
||||||
|
local ui=""; [[ "${QUANTCORE_ENABLE_UI:-true}" != "false" ]] && ui="$QC_API_URL/ui/ (test bench)"
|
||||||
|
step "Ready -- on this Spark:"
|
||||||
|
[[ -n "$ui" ]] && say " $ui"
|
||||||
|
say " $QC_API_URL/docs (API, Swagger)"
|
||||||
|
say " Python: QuantcoreClient(\"$QC_API_URL\") CLI: $0 api health"
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_stop() {
|
||||||
|
need_apptainer
|
||||||
|
local name stopped=0
|
||||||
|
for name in "$API_INSTANCE" "$CUOPT_INSTANCE"; do
|
||||||
|
if instance_running "$name"; then
|
||||||
|
"$APPTAINER" instance stop "$name" >/dev/null && say "stopped $name" && stopped=1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
[[ $stopped == 1 ]] || say "nothing running"
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_status() {
|
||||||
|
need_apptainer
|
||||||
|
step "Instances of $ME"
|
||||||
|
"$APPTAINER" instance list
|
||||||
|
step "cuOpt $CUOPT_URL"
|
||||||
|
local h
|
||||||
|
if h="$(http_get "$CUOPT_URL/cuopt/health")"; then say " up: $h"; else say " not answering"; fi
|
||||||
|
step "API $QC_API_URL"
|
||||||
|
if h="$(http_get "$QC_API_URL/health")"; then
|
||||||
|
if command -v python3 >/dev/null 2>&1; then
|
||||||
|
printf '%s' "$h" | python3 -c '
|
||||||
|
import json, sys
|
||||||
|
h = json.load(sys.stdin); s = h.get("store", {}); c = h.get("cuopt", {})
|
||||||
|
print(" up: version", h.get("version"), "| base currency", h.get("base_currency"))
|
||||||
|
print(" data:", {k: s[k] for k in list(s)[:6]})
|
||||||
|
print(" cuOpt seen by the API:", "reachable" if c.get("reachable") else "NOT reachable -> HiGHS")'
|
||||||
|
else
|
||||||
|
say " up: $h"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
say " not answering"
|
||||||
|
fi
|
||||||
|
step "Files"
|
||||||
|
say " settings: $CONFIG $([[ -f "$CONFIG" ]] || echo '(missing: defaults in use)')"
|
||||||
|
say " data: $QC_DATA ($(du -sh "$QC_DATA" 2>/dev/null | cut -f1 || echo -))"
|
||||||
|
say " quantcore: $(qc_mode) mode, $(qc_image)$([[ $(qc_mode) == venv ]] && echo " + $QC_VENV")"
|
||||||
|
say " cuOpt: $(cuopt_sif)"
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_logs() {
|
||||||
|
local which="${1:-api}" follow=""
|
||||||
|
[[ "${2:-}" == "-f" || "${1:-}" == "-f" ]] && follow=1
|
||||||
|
[[ "$which" == "-f" ]] && which=api
|
||||||
|
local name="$API_INSTANCE"; [[ "$which" == "cuopt" ]] && name="$CUOPT_INSTANCE"
|
||||||
|
local out err
|
||||||
|
out="$(instance_log "$name" out || true)"; err="$(instance_log "$name" err || true)"
|
||||||
|
[[ -n "$out$err" ]] || die "no logs for $name yet"
|
||||||
|
if [[ -n "$follow" ]]; then exec tail -n 50 -F ${out:+"$out"} ${err:+"$err"}; fi
|
||||||
|
tail -n 50 ${out:+"$out"} ${err:+"$err"}
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- data -------------------------------------------------------------------------------------
|
||||||
|
data_job() { # setup|update [args]: through the running API (which then reloads), else in-process
|
||||||
|
local job="$1"; shift
|
||||||
|
if http_get "$QC_API_URL/health" >/dev/null; then
|
||||||
|
say "the API is running: the $job job runs there, and the API reloads the data when it is done"
|
||||||
|
run_qc quantcore-api --url "$QC_API_URL" "$job" --wait "$@"
|
||||||
|
else
|
||||||
|
run_qc quantcore "$job" "$@"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_data_import() {
|
||||||
|
local src="${1:-}" force="${2:-}"
|
||||||
|
[[ -n "$src" ]] || die "usage: $0 data-import <quantcore_data.tar.gz | directory> [--force]"
|
||||||
|
[[ -e "$src" ]] || die "$src not found"
|
||||||
|
mkdir -p "$QC_DATA"
|
||||||
|
if ! data_is_empty && [[ "$force" != "--force" ]]; then
|
||||||
|
die "$QC_DATA already holds data; add --force to overwrite it"
|
||||||
|
fi
|
||||||
|
if [[ -d "$src" ]]; then
|
||||||
|
cp -a "$src/." "$QC_DATA/"
|
||||||
|
else
|
||||||
|
tar -xzf "$src" -C "$QC_DATA"
|
||||||
|
fi
|
||||||
|
say "imported into $QC_DATA ($(du -sh "$QC_DATA" | cut -f1))"
|
||||||
|
if instance_running "$API_INSTANCE"; then say "the running API still holds the old data: $0 restart"; fi
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_data_export() {
|
||||||
|
local dst="${1:-quantcore_data.tar.gz}"
|
||||||
|
[[ "$dst" = /* ]] || dst="$PWD/$dst"
|
||||||
|
tar -czf "$dst" -C "$QC_DATA" --exclude './jobs' --exclude '*.log' .
|
||||||
|
say "wrote $dst ($(du -h "$dst" | cut -f1))"
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_env() {
|
||||||
|
cat <<EOF
|
||||||
|
QC_HOME=$QC_HOME
|
||||||
|
config=$CONFIG $([[ -f "$CONFIG" ]] || echo '(missing)')
|
||||||
|
QC_DATA=$QC_DATA
|
||||||
|
QC_IMAGES=$QC_IMAGES
|
||||||
|
quantcore=$(qc_mode) mode: $(qc_image) (QC_IMAGE=$QC_IMAGE)
|
||||||
|
cuOpt image=$(cuopt_sif) (QC_CUOPT_IMAGE=$QC_CUOPT_IMAGE)
|
||||||
|
API=$QC_API_URL
|
||||||
|
cuOpt=$CUOPT_URL (QC_CUOPT_START=$QC_CUOPT_START)
|
||||||
|
EOF
|
||||||
|
compgen -v | grep '^QUANTCORE_' | while read -r v; do say "$v=${!v}"; done
|
||||||
|
}
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<EOF
|
||||||
|
quantcore on this DGX Spark (Apptainer, localhost only). Settings: $CONFIG
|
||||||
|
|
||||||
|
Setup
|
||||||
|
install [--force] config, cuOpt image (pull), quantcore image (build or prebuilt), checks
|
||||||
|
build rebuild quantcore from this repository (after a code change / git pull)
|
||||||
|
pull-cuopt [--force] download the cuOpt image again
|
||||||
|
data-import SRC [--force] copy a prepared data set (.tar.gz or directory) into $QC_DATA
|
||||||
|
data-export [FILE] pack the data directory (default quantcore_data.tar.gz)
|
||||||
|
setup [--offline] build the data set from Yahoo Finance (~15 min, needs internet)
|
||||||
|
update [--full] fetch the newest prices / FX
|
||||||
|
Run
|
||||||
|
start | stop | restart cuOpt + API instances
|
||||||
|
status instances, health, data, images
|
||||||
|
logs [api|cuopt] [-f] instance logs
|
||||||
|
Use
|
||||||
|
cli COMMAND ... quantcore CLI in the container: status, risk, performance, optimize, snapshot ...
|
||||||
|
api COMMAND ... quantcore-api against $QC_API_URL: health, selftest, optimize ...
|
||||||
|
python [ARGS] Python with quantcore installed (QUANTCORE_* set as for the API)
|
||||||
|
shell a shell in the quantcore container
|
||||||
|
env the effective settings
|
||||||
|
clean-cache remove Apptainer's download cache (layers of pulled images)
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
main() {
|
||||||
|
local cmd="${1:-help}"; [[ $# -gt 0 ]] && shift
|
||||||
|
case "$cmd" in
|
||||||
|
install) cmd_install "$@" ;;
|
||||||
|
build) cmd_build ;;
|
||||||
|
pull-cuopt) cmd_pull_cuopt "$@" ;;
|
||||||
|
start) cmd_start ;;
|
||||||
|
stop) cmd_stop ;;
|
||||||
|
restart) cmd_stop; cmd_start ;;
|
||||||
|
status) cmd_status ;;
|
||||||
|
logs) cmd_logs "$@" ;;
|
||||||
|
cli) run_qc quantcore "$@" ;;
|
||||||
|
api) run_qc quantcore-api --url "$QC_API_URL" "$@" ;;
|
||||||
|
python) run_qc python "$@" ;;
|
||||||
|
shell) need_apptainer; export_quantcore_env; mapfile -t binds < <(qc_binds)
|
||||||
|
exec "$APPTAINER" shell --cleanenv "${binds[@]}" "$(qc_image)" ;;
|
||||||
|
setup|update) data_job "$cmd" "$@" ;;
|
||||||
|
data-import) cmd_data_import "$@" ;;
|
||||||
|
data-export) cmd_data_export "$@" ;;
|
||||||
|
env) cmd_env ;;
|
||||||
|
clean-cache) need_apptainer; "$APPTAINER" cache clean --force ;;
|
||||||
|
help|-h|--help) usage ;;
|
||||||
|
*) usage; die "unknown command: $cmd" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
75
deploy/apptainer/quantcore.def
Normal file
75
deploy/apptainer/quantcore.def
Normal file
@ -0,0 +1,75 @@
|
|||||||
|
# quantcore as an Apptainer image -- for a DGX Spark (aarch64) where users have no root and no Docker.
|
||||||
|
#
|
||||||
|
# Build from the repository root (paths under %files are relative to the current directory):
|
||||||
|
# apptainer build quantcore.sif deploy/apptainer/quantcore.def
|
||||||
|
# Without root, Apptainer builds through --fakeroot automatically; deploy/apptainer/qc.sh build does
|
||||||
|
# exactly this. The image contains Python, the quantcore package and HiGHS (inside SciPy); cuOpt is a
|
||||||
|
# separate container (deploy/apptainer/qc.sh pull-cuopt) reached at http://127.0.0.1:5000.
|
||||||
|
#
|
||||||
|
# Use it through deploy/apptainer/qc.sh (start/stop/status/cli), or directly:
|
||||||
|
# apptainer run --bind ~/quantcore/data:/data quantcore.sif status # the quantcore CLI
|
||||||
|
# apptainer instance start --bind ~/quantcore/data:/data quantcore.sif quantcore-api # API + test bench
|
||||||
|
Bootstrap: docker
|
||||||
|
From: python:3.12-slim
|
||||||
|
|
||||||
|
%labels
|
||||||
|
org.opencontainers.image.title quantcore
|
||||||
|
org.opencontainers.image.version 0.1.0
|
||||||
|
org.opencontainers.image.description Deterministic quant core of the AISE502 portfolio platform: data, risk, Black-Litterman, mean-CVaR MILP (cuOpt / HiGHS)
|
||||||
|
|
||||||
|
%help
|
||||||
|
quantcore -- portfolio data, risk measurement, Black-Litterman and mean-CVaR MILP optimisation.
|
||||||
|
|
||||||
|
The runscript is the quantcore CLI: apptainer run quantcore.sif <command> [options]
|
||||||
|
The startscript serves the HTTP API: apptainer instance start quantcore.sif quantcore-api
|
||||||
|
-> http://127.0.0.1:5001/ui/ (test bench) and /docs (Swagger); bound to localhost only.
|
||||||
|
|
||||||
|
Bind a writable data directory to /data (QUANTCORE_DATA_DIR). cuOpt is expected at
|
||||||
|
QUANTCORE_CUOPT_URL (default http://127.0.0.1:5000); without it every optimisation uses HiGHS.
|
||||||
|
All QUANTCORE_* variables of quantcore/config.py can be passed with --env.
|
||||||
|
|
||||||
|
%files
|
||||||
|
pyproject.toml /opt/quantcore-src/pyproject.toml
|
||||||
|
README.md /opt/quantcore-src/README.md
|
||||||
|
quantcore /opt/quantcore-src/quantcore
|
||||||
|
deploy/apptainer/constraints.txt /opt/quantcore-src/constraints.txt
|
||||||
|
|
||||||
|
%post
|
||||||
|
set -eu
|
||||||
|
export PIP_NO_CACHE_DIR=1 PIP_DISABLE_PIP_VERSION_CHECK=1 PIP_ROOT_USER_ACTION=ignore
|
||||||
|
# constraints.txt pins the library versions the image was verified with (SciPy decides the HiGHS
|
||||||
|
# version, and with it which of several equally good integer solutions HiGHS returns).
|
||||||
|
python -m pip install -c /opt/quantcore-src/constraints.txt /opt/quantcore-src
|
||||||
|
rm -rf /opt/quantcore-src
|
||||||
|
find /usr/local/lib -name '__pycache__' -type d -prune -exec rm -rf {} + 2>/dev/null || true
|
||||||
|
mkdir -p /data
|
||||||
|
python -c "import quantcore, scipy; print('quantcore', quantcore.__version__, '| scipy', scipy.__version__)"
|
||||||
|
|
||||||
|
%environment
|
||||||
|
export PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
|
||||||
|
# a student's own pip --user packages must not shadow the image's libraries
|
||||||
|
export PYTHONNOUSERSITE=1
|
||||||
|
unset PYTHONPATH PYTHONHOME VIRTUAL_ENV
|
||||||
|
# small-matrix linear algebra (the innovation filter) is ~50x slower with all 20 cores
|
||||||
|
export OPENBLAS_NUM_THREADS="${OPENBLAS_NUM_THREADS:-4}" OMP_NUM_THREADS="${OMP_NUM_THREADS:-4}"
|
||||||
|
export MKL_NUM_THREADS="${MKL_NUM_THREADS:-4}"
|
||||||
|
export QUANTCORE_DATA_DIR="${QUANTCORE_DATA_DIR:-/data}"
|
||||||
|
export QUANTCORE_CUOPT_URL="${QUANTCORE_CUOPT_URL:-http://127.0.0.1:5000}"
|
||||||
|
export QUANTCORE_API_HOST="${QUANTCORE_API_HOST:-127.0.0.1}"
|
||||||
|
export QUANTCORE_API_PORT="${QUANTCORE_API_PORT:-5001}"
|
||||||
|
|
||||||
|
%runscript
|
||||||
|
exec quantcore "$@"
|
||||||
|
|
||||||
|
%startscript
|
||||||
|
exec quantcore serve
|
||||||
|
|
||||||
|
%test
|
||||||
|
set -eu
|
||||||
|
python - <<'PY'
|
||||||
|
import importlib.resources as r
|
||||||
|
import quantcore, quantcore.api.app as app # imports FastAPI app, all services and the UI mount
|
||||||
|
from scipy.optimize import milp # HiGHS, the CPU fallback solver
|
||||||
|
assert (r.files("quantcore.api") / "ui" / "index.html").is_file(), "test bench missing from the wheel"
|
||||||
|
print("quantcore", quantcore.__version__, "ok; UI mounted:", app.UI_ENABLED)
|
||||||
|
PY
|
||||||
40
deploy/apptainer/quantcore.env.example
Normal file
40
deploy/apptainer/quantcore.env.example
Normal file
@ -0,0 +1,40 @@
|
|||||||
|
# Settings of deploy/apptainer/qc.sh -- quantcore standalone on a DGX Spark with Apptainer.
|
||||||
|
# `qc.sh install` copies this file to $QC_HOME/quantcore.env (QC_HOME defaults to ~/quantcore).
|
||||||
|
# qc.sh reads it on every call; after a change run `qc.sh restart`. A variable that is also set in
|
||||||
|
# the shell environment wins over this file (e.g. `QC_API_PORT=5011 qc.sh start`).
|
||||||
|
|
||||||
|
# --- where things live ------------------------------------------------------------------------
|
||||||
|
#QC_DATA=$HOME/quantcore/data # market data: universe, prices, FX, snapshots (~40 MB)
|
||||||
|
#QC_IMAGES=$HOME/quantcore/images # quantcore.sif (~0.2 GB) and cuopt.sif (~3.3 GB)
|
||||||
|
|
||||||
|
# --- network: nothing leaves this machine -----------------------------------------------------
|
||||||
|
# Both servers listen on 127.0.0.1 only. Ports are shared by every user of this Spark: if
|
||||||
|
# `qc.sh start` reports a port as taken, pick another one here (it suggests a free one).
|
||||||
|
QC_API_PORT=5001 # quantcore API + test bench: http://127.0.0.1:5001/ui/
|
||||||
|
QC_CUOPT_PORT=5000 # cuOpt server
|
||||||
|
|
||||||
|
# --- cuOpt: the GPU MILP solver, in its own container -----------------------------------------
|
||||||
|
# auto: use a cuOpt that already answers on QC_CUOPT_PORT (another user's or a shared one),
|
||||||
|
# otherwise start our own instance on the GPU
|
||||||
|
# yes: always start our own instance (fails if the port is taken)
|
||||||
|
# no: no cuOpt -- every optimisation runs on HiGHS (CPU, same model, same answer)
|
||||||
|
QC_CUOPT_START=auto
|
||||||
|
# Official NVIDIA image (arm64 + amd64), pulled once without root. If the Spark already provides a
|
||||||
|
# cuOpt image, put the path of its .sif file here instead.
|
||||||
|
QC_CUOPT_IMAGE=docker://nvidia/cuopt:26.8.0-cuda13.3-py3.14
|
||||||
|
|
||||||
|
# --- the quantcore image ----------------------------------------------------------------------
|
||||||
|
# build: build it from this repository (Apptainer's unprivileged --fakeroot build); if this
|
||||||
|
# Spark does not allow builds, install falls back to 'venv' by itself
|
||||||
|
# venv: no build: a plain Python image + a virtualenv in $QC_HOME/venv (same libraries)
|
||||||
|
# /path/x.sif: a prebuilt image (e.g. handed out by the lecturer) -- used in place, not copied
|
||||||
|
# docker://... or oras://... to pull a published one
|
||||||
|
QC_IMAGE=build
|
||||||
|
|
||||||
|
# --- passed unchanged into the quantcore container (see quantcore/config.py) -------------------
|
||||||
|
QUANTCORE_CUOPT_TIME_LIMIT=300 # seconds per MILP
|
||||||
|
QUANTCORE_ENABLE_UI=true # false: JSON API and /docs only, no test bench at /ui/
|
||||||
|
QUANTCORE_LOG_LEVEL=INFO
|
||||||
|
#QUANTCORE_BASE_CURRENCY=CHF
|
||||||
|
#QUANTCORE_CUOPT_URL=http://127.0.0.1:5000 # set only to use a cuOpt at another address; then
|
||||||
|
# qc.sh never starts its own
|
||||||
57
deploy/deploy_to_spark.sh
Executable file
57
deploy/deploy_to_spark.sh
Executable file
@ -0,0 +1,57 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Deploy the quant service to the DGX Spark:
|
||||||
|
# 1. rsync the repository to ~/services/work/Portfolio_Risk_Optimization
|
||||||
|
# 2. optionally rsync an already built data directory (saves the 15-minute setup on the Spark)
|
||||||
|
# 3. add the `quantcore` service block to ~/services/docker-compose.yml (idempotent)
|
||||||
|
# 4. build the image and start the container
|
||||||
|
#
|
||||||
|
# Usage: deploy/deploy_to_spark.sh [--with-data] [--host user_fh_fhgr@spark-6d4a]
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
HOST="user_fh_fhgr@spark-6d4a"
|
||||||
|
WITH_DATA=0
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in
|
||||||
|
--with-data) WITH_DATA=1 ;;
|
||||||
|
--host=*) HOST="${arg#--host=}" ;;
|
||||||
|
--host) shift; HOST="$1" ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
REPO_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
REMOTE_SERVICES="services"
|
||||||
|
REMOTE_REPO="$REMOTE_SERVICES/work/Portfolio_Risk_Optimization"
|
||||||
|
REMOTE_DATA="$REMOTE_SERVICES/work/quantcore_data"
|
||||||
|
|
||||||
|
echo ">> rsync repository -> $HOST:$REMOTE_REPO"
|
||||||
|
ssh "$HOST" "mkdir -p $REMOTE_REPO $REMOTE_DATA"
|
||||||
|
rsync -az --delete \
|
||||||
|
--exclude '.git' --exclude '/data/' --exclude '__pycache__' --exclude '*.pyc' --exclude '.pytest_cache' \
|
||||||
|
--exclude '*.egg-info' --exclude '.ruff_cache' --exclude 'scratch/' \
|
||||||
|
"$REPO_DIR/" "$HOST:$REMOTE_REPO/"
|
||||||
|
|
||||||
|
if [[ "$WITH_DATA" == "1" && -d "$REPO_DIR/data" ]]; then
|
||||||
|
echo ">> rsync data directory -> $HOST:$REMOTE_DATA"
|
||||||
|
rsync -az --exclude 'jobs/' --exclude '*.log' "$REPO_DIR/data/" "$HOST:$REMOTE_DATA/"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ">> register the service in ~/services/docker-compose.yml (if missing)"
|
||||||
|
ssh "$HOST" bash -s <<'REMOTE'
|
||||||
|
set -euo pipefail
|
||||||
|
cd ~/services
|
||||||
|
if ! grep -q '^ quantcore:' docker-compose.yml; then
|
||||||
|
cp docker-compose.yml "docker-compose.yml.bak.$(date +%Y%m%d%H%M%S)"
|
||||||
|
# append the service block (everything below "services:" of the snippet)
|
||||||
|
awk 'f{print} /^services:/{f=1}' work/Portfolio_Risk_Optimization/deploy/docker-compose.quantcore.yml >> docker-compose.yml
|
||||||
|
echo " added quantcore service block"
|
||||||
|
else
|
||||||
|
echo " quantcore service already present"
|
||||||
|
fi
|
||||||
|
docker compose build quantcore
|
||||||
|
docker compose up -d quantcore
|
||||||
|
sleep 3
|
||||||
|
docker compose ps quantcore
|
||||||
|
curl -s http://localhost:5001/health | head -c 600; echo
|
||||||
|
REMOTE
|
||||||
|
|
||||||
|
echo ">> done. API: http://spark-6d4a:5001/docs"
|
||||||
26
deploy/docker-compose.quantcore.yml
Normal file
26
deploy/docker-compose.quantcore.yml
Normal file
@ -0,0 +1,26 @@
|
|||||||
|
# Service block for the DGX Spark's ~/services/docker-compose.yml (deploy/deploy_to_spark.sh
|
||||||
|
# appends it). Standalone use: docker compose -f deploy/docker-compose.quantcore.yml up -d
|
||||||
|
#
|
||||||
|
# The service reaches cuOpt through the compose network (http://cuopt:5000) and keeps all its
|
||||||
|
# data under ~/services/work/quantcore_data on the host (survives rebuilds and restarts).
|
||||||
|
services:
|
||||||
|
quantcore:
|
||||||
|
build:
|
||||||
|
context: ./work/Portfolio_Risk_Optimization # the rsynced repository
|
||||||
|
dockerfile: deploy/Dockerfile
|
||||||
|
image: local/quantcore:0.1
|
||||||
|
container_name: quantcore
|
||||||
|
restart: unless-stopped
|
||||||
|
ports: ["5001:5001"]
|
||||||
|
environment:
|
||||||
|
QUANTCORE_DATA_DIR: /data
|
||||||
|
QUANTCORE_CUOPT_URL: http://cuopt:5000
|
||||||
|
QUANTCORE_CUOPT_TIME_LIMIT: "300"
|
||||||
|
QUANTCORE_LOG_LEVEL: INFO
|
||||||
|
QUANTCORE_ENABLE_UI: "true" # "false" for an installation whose users bring their own front end
|
||||||
|
OPENBLAS_NUM_THREADS: "4" # small-matrix linear algebra is much slower with 20 threads
|
||||||
|
OMP_NUM_THREADS: "4"
|
||||||
|
volumes:
|
||||||
|
- ./work/quantcore_data:/data
|
||||||
|
depends_on:
|
||||||
|
- cuopt
|
||||||
30
deploy/docker-compose.students.yml
Normal file
30
deploy/docker-compose.students.yml
Normal file
@ -0,0 +1,30 @@
|
|||||||
|
# quantcore for a platform team: the JSON API and /docs, no test bench.
|
||||||
|
#
|
||||||
|
# docker compose -f deploy/docker-compose.students.yml up -d --build
|
||||||
|
# curl http://localhost:5001/health
|
||||||
|
#
|
||||||
|
# The MILP solver is optional. Point QUANTCORE_CUOPT_URL at a cuOpt server if you have one (on a DGX
|
||||||
|
# Spark: http://<host>:5000); if it is unreachable, every optimisation falls back to HiGHS on the CPU
|
||||||
|
# -- slower on large problems, same model, same answer, and the result says which solver produced it.
|
||||||
|
#
|
||||||
|
# The data directory is mounted, not baked in: run the setup once (POST /jobs/setup, ~15 min) or copy
|
||||||
|
# a prepared ./data next to this file.
|
||||||
|
services:
|
||||||
|
quantcore:
|
||||||
|
build:
|
||||||
|
context: ..
|
||||||
|
dockerfile: deploy/Dockerfile
|
||||||
|
image: quantcore:0.1
|
||||||
|
container_name: quantcore
|
||||||
|
restart: unless-stopped
|
||||||
|
ports: ["5001:5001"]
|
||||||
|
environment:
|
||||||
|
QUANTCORE_DATA_DIR: /data
|
||||||
|
QUANTCORE_CUOPT_URL: ${QUANTCORE_CUOPT_URL:-http://cuopt:5000}
|
||||||
|
QUANTCORE_CUOPT_TIME_LIMIT: "300"
|
||||||
|
QUANTCORE_ENABLE_UI: "false" # the test bench is not part of the deliverable
|
||||||
|
QUANTCORE_LOG_LEVEL: INFO
|
||||||
|
OPENBLAS_NUM_THREADS: "4" # small-matrix linear algebra is much slower with all cores
|
||||||
|
OMP_NUM_THREADS: "4"
|
||||||
|
volumes:
|
||||||
|
- ../data:/data
|
||||||
125
docs/ARCHITECTURE.md
Normal file
125
docs/ARCHITECTURE.md
Normal file
@ -0,0 +1,125 @@
|
|||||||
|
# Architecture
|
||||||
|
|
||||||
|
`quantcore` is the **deterministic core** of the platform sketched in the AISE502 project brief:
|
||||||
|
a set of services with plain contracts, no LLM inside, resilient against external data outages,
|
||||||
|
and reproducible through pinned data snapshots. The AI layer (news ingestion, `ResearchAgent`,
|
||||||
|
`AdvisorAgent`, …) sits *on top* and only reads these services through their contracts.
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────────────────────────────┐
|
||||||
|
HTTP/JSON │ quantcore.api (FastAPI) – thin, no logic, background jobs │
|
||||||
|
└───────┬───────────────┬───────────────┬──────────────┬───────────┘
|
||||||
|
│ │ │ │
|
||||||
|
┌─────────────────▼──┐ ┌─────────▼─────────┐ ┌──▼──────────┐ ┌▼────────────────────┐
|
||||||
|
│ SetupService │ │ RiskService │ │ Performance │ │ OptimizationService │
|
||||||
|
│ DataDownloadService │ │ (measurement.risk) │ │ Service │ │ (optimization) │
|
||||||
|
└─────────┬───────────┘ └─────────┬─────────┘ └──┬──────────┘ └┬───────────┬────────┘
|
||||||
|
│ │ │ │ │
|
||||||
|
┌─────────▼───────────┐ ┌─────────▼───────────────▼──────────┐ │ ┌───────▼────────────┐
|
||||||
|
│ YahooClient (ext.) │ │ MarketDataService │◄──┘ │ models: │
|
||||||
|
│ timeout · retry · │ │ prices · FX → CHF · returns · │ │ CovarianceModel │
|
||||||
|
│ circuit breaker │ │ market caps · cap-weighted index │ │ GBM / innovations │
|
||||||
|
└─────────┬───────────┘ └─────────┬──────────────────────────┘ │ Black-Litterman │
|
||||||
|
│ │ └───────┬────────────┘
|
||||||
|
┌─────────▼────────────────────────▼──────────┐ ┌────────────────▼────────────┐
|
||||||
|
│ DataStore (parquet/csv under data_dir, │ │ MeanCVaRProblem → MILPModel │
|
||||||
|
│ manifest, snapshots) │ │ CuOptSolver ⇄ HighsSolver │
|
||||||
|
└─────────────────────────────────────────────┘ └──────────────┬──────────────┘
|
||||||
|
│ REST
|
||||||
|
┌─────────────▼─────────────┐
|
||||||
|
│ cuOpt server (DGX Spark) │
|
||||||
|
└───────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Bounded contexts → modules
|
||||||
|
|
||||||
|
| Context | Module | Contract (inputs → outputs) |
|
||||||
|
|---|---|---|
|
||||||
|
| Universe | `quantcore.universe` | web candidate lists → `universe.csv` (symbol, name, region, country, currency, exchange, GICS sector, Yahoo sector/industry, market cap local & CHF, shares outstanding, first price date, coverage) |
|
||||||
|
| Market data | `quantcore.data` | Yahoo → `adj_close.parquet` (date × symbol, listing currency), `fx_chf.parquet` (date × currency); `MarketDataService.prices/returns(symbols, start, end, currency, freq)` → aligned frames |
|
||||||
|
| Measurement | `quantcore.measurement` | returns frame → risk tables, series, matrices, `PortfolioRisk`; performance tables |
|
||||||
|
| Models | `quantcore.models` | returns frame → covariance, `ScenarioSet` (S × N horizon log-returns, optional probabilities, fingerprint; `sampling`: Sobol'/moment matching/importance sampling), `BlackLittermanResult` |
|
||||||
|
| Optimisation | `quantcore.optimization` | `OptimizationRequest` → `OptimizationResult` (shares, trades, cash, costs, CVaR, exposures, provenance of every model input, scenario hash, reproducibility level); `problem` (MILP, weighted scenarios or Gaussian cuts), `rowgen` (tail-row generation), `gaussian_exact` (outer approximation), `solvers` |
|
||||||
|
| Interface | `quantcore.api`, `quantcore.cli` | JSON/CLI surfaces of the above; no computation of their own |
|
||||||
|
| Test bench | `quantcore/api/ui` | a static single-page client of the JSON API (`/ui/`): stock data, covariance and scenarios, views, constraints, optimisation with before/after. Static HTML + ES modules, no build step and no CDN, so it also runs in the offline container. It formats and charts what the services return and computes nothing itself |
|
||||||
|
|
||||||
|
Dependencies point downwards only: `api → optimization → models → measurement → data → config`.
|
||||||
|
`measurement` and `models` never touch storage or the network; they operate on DataFrames handed
|
||||||
|
to them, which is what makes them pure and testable with reference vectors.
|
||||||
|
|
||||||
|
## Deterministic by construction
|
||||||
|
|
||||||
|
* Every stochastic step takes an explicit `seed` (scenario generation: scrambled Sobol' points,
|
||||||
|
moment matched — the same request yields the same scenario matrix, fingerprinted as
|
||||||
|
`scenario_hash`; bit-identical within one environment, while across environments the *key* matches
|
||||||
|
and the digest need not (§9 of `docs/METHODOLOGY.md` measures why); the *scenario key* lists
|
||||||
|
exactly what it depends on — risk limits, constraints,
|
||||||
|
costs, objective, cash, holdings and views are not part of it, so a family of runs is compared on
|
||||||
|
one set, which the service also caches) and the same portfolio with HiGHS (`reproducibility="exact"`); on cuOpt the
|
||||||
|
figures agree within its MIP gap of 1e-4 (`"within_gap"`). `scenario_model="gaussian_exact"` needs
|
||||||
|
no scenarios at all.
|
||||||
|
* Every result carries the *provenance* of its inputs: covariance model, scenario model and seed,
|
||||||
|
expected-return model (incl. the views and their calibrated Ω), constraints, costs, solver, status,
|
||||||
|
gap, timings. An advisor can cite “CVaR 95 % = CHF 38 210 (OptimizationService, empirical model,
|
||||||
|
EWM λ=0.99, seed 42)”.
|
||||||
|
* Pinned snapshots (`DataStore.create_snapshot`, `MarketDataService(settings, snapshot=...)`) freeze
|
||||||
|
universe, prices and FX so that demos and grading are reproducible even if Yahoo misbehaves.
|
||||||
|
|
||||||
|
## Where the solving happens
|
||||||
|
|
||||||
|
The MILP is built in `quantcore.optimization.problem` and handed to one of two solvers, both fed the
|
||||||
|
same `MILPModel`:
|
||||||
|
|
||||||
|
* **cuOpt** on the GPU, over HTTP to the cuOpt container on the same Spark (`QUANTCORE_CUOPT_URL`):
|
||||||
|
`http://127.0.0.1:5000` in the Apptainer setup, where both containers share the host's network,
|
||||||
|
and `http://cuopt:5000` inside the Docker compose network. The only GPU component; an external
|
||||||
|
dependency, hence the timeouts, retries and circuit breaker.
|
||||||
|
* **HiGHS** on the CPU, *in the same process as the service* — it is compiled into SciPy
|
||||||
|
(`scipy.optimize.milp`), so there is nothing to deploy or start, and no network hop. In production
|
||||||
|
that means inside the `quantcore` container (SciPy 1.18.1 → HiGHS 1.12.0, 4 threads).
|
||||||
|
|
||||||
|
`solver="auto"` tries cuOpt and falls back to HiGHS; the result names the solver that produced it and
|
||||||
|
how reproducible it is. See `docs/METHODOLOGY.md` §9 for versions and the measured comparison.
|
||||||
|
|
||||||
|
## Resilience on every external call
|
||||||
|
|
||||||
|
| External dependency | Pattern | Fallback |
|
||||||
|
|---|---|---|
|
||||||
|
| Yahoo Finance (`YahooClient`) | per-request timeout, 4 retries with exponential back-off, circuit breaker (5 failures → 60 s open) | the stored data / a snapshot; the API keeps serving from the store |
|
||||||
|
| Candidate lists (Wikipedia, iShares) | timeout, one source after the other | static snapshots shipped in `quantcore/data/static/` |
|
||||||
|
| cuOpt (`CuOptSolver`) | submit with retry/back-off, polling with client-side deadline, circuit breaker, results deleted server-side | local HiGHS MILP (scipy) — the result says `solver: "highs"` and carries a warning |
|
||||||
|
|
||||||
|
## Data model conventions
|
||||||
|
|
||||||
|
* prices: adjusted close (splits & dividends), one column per Yahoo symbol, union calendar of all
|
||||||
|
exchanges, holidays forward-filled (≤ 5 days); pence-quoted London lines converted to GBP.
|
||||||
|
* FX: cross rates against the base currency, repaired for quoting-unit changes (`clean_fx`) —
|
||||||
|
a rate returned 100× off for a stretch of days would otherwise be read as a 100× move of
|
||||||
|
every stock in that currency (see `docs/METHODOLOGY.md` §1).
|
||||||
|
* returns: natural logarithms; CHF return = local return + FX return (additive).
|
||||||
|
* frequencies: daily, weekly (Friday close), monthly (month end); horizons: day = 1, week = 5,
|
||||||
|
month = 21 trading days (or 1 / 4 weeks at weekly frequency).
|
||||||
|
* money: everything in the base currency (CHF); positions are **integer share numbers**.
|
||||||
|
|
||||||
|
## Where the AI layer plugs in
|
||||||
|
|
||||||
|
* `ResearchAgent` insights (ticker, sentiment ∈ [−1, 1], degree of belief) become Black–Litterman
|
||||||
|
views through `view_from_sentiment` / the `views` field of `/optimize` — the numbers (view return,
|
||||||
|
Ω, posterior) are computed here, the agent only supplies the assessment and its provenance.
|
||||||
|
* `RiskAgent` / `OptimizationAgent` call `/risk/*` and `/optimize` and *explain* the returned figures.
|
||||||
|
* The ontology guard can validate tickers and sectors against `/universe`.
|
||||||
|
|
||||||
|
## Judging the system by hand
|
||||||
|
|
||||||
|
The test bench at `/ui/` exists so that every model choice can be exercised without writing code:
|
||||||
|
the same scenario set can be seen as a distribution, as per-stock moments and as the risk figures of
|
||||||
|
an optimised portfolio, and a run's provenance (scenario key and hash, solver, gap, reproducibility)
|
||||||
|
is shown next to its result. It is also the quickest way to see the invariants hold — two identical
|
||||||
|
requests share a scenario hash and produce identical shares, and a family of runs that varies only
|
||||||
|
the risk limit or the constraints reuses one scenario set.
|
||||||
|
|
||||||
|
## Operations
|
||||||
|
|
||||||
|
* `quantcore setup` (once), `quantcore update` (scheduled, e.g. nightly), `quantcore snapshot`.
|
||||||
|
* On the Spark the API runs as the `quantcore` container next to `cuopt`, `gurobi`, `jupyter`
|
||||||
|
(see `docs/SPARK_DEPLOYMENT.md`); data lives in `~/services/work/quantcore_data`.
|
||||||
437
docs/METHODOLOGY.md
Normal file
437
docs/METHODOLOGY.md
Normal file
@ -0,0 +1,437 @@
|
|||||||
|
# Methodology
|
||||||
|
|
||||||
|
Every formula the services implement, with the conventions and default parameters. Section
|
||||||
|
numbers follow the project brief ("Portfolio Risk Measurement and Optimization").
|
||||||
|
|
||||||
|
## 1. Investment universe (`quantcore.universe`)
|
||||||
|
|
||||||
|
* **Candidates.** US: the S&P 500 constituents (Wikipedia). Europe (EU, CH, NO, UK): the STOXX
|
||||||
|
Europe 600 constituents, read from the iShares STOXX Europe 600 UCITS ETF holdings file, which
|
||||||
|
gives issuer ticker, exchange, listing currency, country and GICS sector. Domiciles outside
|
||||||
|
EU/CH/NO/UK (e.g. Bermuda, Israel) are excluded. Both lists are snapshotted into
|
||||||
|
`quantcore/data/static/` for offline use.
|
||||||
|
* **Yahoo symbols.** `<code><suffix>` per exchange (`.SW .L .DE .PA .AS .MI .MC .ST .CO .OL .HE
|
||||||
|
.BR .VI .WA .IR .LS .AT`), share classes with a dash (`NOVO-B.CO`, `BRK-B`), London codes lose
|
||||||
|
the trailing dot (`RR.` → `RR.L`). All 1 117 symbols were verified against Yahoo, and the
|
||||||
|
European prices were cross-checked against the ETF file (after FX) — zero mismatches.
|
||||||
|
* **Metadata** from `Ticker.info`: long name, domicile, listing currency (GBp → GBP with a price
|
||||||
|
scale of 100), exchange, Yahoo sector/industry, market cap, shares outstanding, first trade date.
|
||||||
|
* **Selection.** One listing per company (share classes collapsed on a normalised name, largest cap
|
||||||
|
kept) → rank by market cap in CHF within each region → keep the largest 500 US and 500 European
|
||||||
|
→ drop stocks whose first price is after `required_history_from` (2005-01-31) or whose coverage
|
||||||
|
since then is below 90 % of union-calendar days. The result has fewer than 1 000 names.
|
||||||
|
* **Market capitalisation** (for index weights) at any date: shares outstanding (today's, held
|
||||||
|
constant) × adjusted close × FX to CHF.
|
||||||
|
* **Price cleaning** (`quantcore.data.cleaning`, applied by the setup and download services and
|
||||||
|
re-runnable with `quantcore clean`): non-positive adjusted closes (broken dividend/rights
|
||||||
|
adjustments, e.g. ABI.BR before 2008) → NaN; *spike blocks* — at most three days whose entry
|
||||||
|
move `|r| > 0.4` is reversed within the block (stale pre-split ticks on exchange holidays, e.g.
|
||||||
|
2× ticks in NOVO-B.CO) → NaN. Genuine jumps are not reversed and therefore untouched. Stocks
|
||||||
|
whose cleaned series has a gap longer than `max_missing_run` (20 business days) after
|
||||||
|
2005-01-31 are excluded from the universe. On the 2026-09-21 data: 1 347 non-positive values in
|
||||||
|
2 symbols, 28 spike ticks in 8 symbols; final universe 745 stocks (382 US, 363 European).
|
||||||
|
* **FX quoting-unit repair** (`clean_fx`, same module and the same two entry points): Yahoo returns
|
||||||
|
a cross rate in the wrong unit for whole stretches of days. NOK/CHF came back **100× too small on
|
||||||
|
96 observations** — 2016-08-15 to 2017-01-09 in three runs, plus 2022-11-04 — which put the
|
||||||
|
measured volatility of the *currency* at 337 % a year instead of 12 %, and with it every Norwegian
|
||||||
|
stock (a CHF return is the local return plus the FX return, so EQNR.OL measured 338 % against its
|
||||||
|
true 31 %, and its Black–Litterman implied return came out above 100 % a year). A day-on-day
|
||||||
|
log-change within 0.15 of `k·ln(10)` is treated as a unit change, accumulated, and the series is
|
||||||
|
put back on the scale that most of its observations share (ties go to the scale it ends on).
|
||||||
|
The tolerance is tight on purpose: a unit change is *exactly* a power of ten apart from the day's
|
||||||
|
own move, while a loose tolerance swallows real events — at 0.5 it flagged ABVX.PA's +510 %
|
||||||
|
trial-result day (ratio 6.1) as a decimal shift. A scan of all 1 117 price series with the tight
|
||||||
|
rule finds one further case, ROSE.L (11 observations quoted in pence rather than GBP), which is
|
||||||
|
not in the investable universe.
|
||||||
|
* **FX spike ticks** (same function, thresholds an order of magnitude tighter than for prices:
|
||||||
|
entry move `|r| > 0.05` reversed within 0.02 over at most three days). Five ticks in twenty-one
|
||||||
|
years: EUR, GBP and USD on 2009-02-06 (one bad CHF quote hits every cross that day), SEK on
|
||||||
|
2010-09-13, NOK on 2022-10-10. Removing them takes the lag-1 autocorrelation of daily FX
|
||||||
|
log-returns to about zero — EUR −0.053 → −0.005, GBP −0.040 → −0.001, USD −0.034 → −0.002 — which
|
||||||
|
is what a clean daily close series looks like. The one violent day in the sample, 2015-01-16
|
||||||
|
(the SNB abandons the euro floor, every CHF cross moves ~18 %), does not reverse and is kept.
|
||||||
|
|
||||||
|
**Why the rates are taken directly against CHF and not crossed through USD.** USD pairs are the most
|
||||||
|
liquid instruments, so triangulating `CUR/CHF = (USD/CHF) / (USD/CUR)` looks attractive; measured on
|
||||||
|
this feed it is worse, and the package therefore uses the direct pair wherever it has history (the
|
||||||
|
USD cross remains the fallback, and is what PLN uses, since `PLNCHF=X` only starts in 2026):
|
||||||
|
|
||||||
|
| | direct `CURCHF=X` | crossed through USD |
|
||||||
|
|---|---|---|
|
||||||
|
| EUR volatility p.a. / lag-1 AC | 7.49 % / −0.005 | 9.99 % / **−0.219** |
|
||||||
|
| NOK volatility p.a. / lag-1 AC | 11.45 % / −0.047 | 17.14 % / **−0.330** |
|
||||||
|
| GBP volatility p.a. | 10.10 % | 10.35 % |
|
||||||
|
| reversing bad ticks, 2005–2026 | 5 over all currencies | 8, five of them in `EUR=X` alone |
|
||||||
|
|
||||||
|
The cross combines two series snapped at different times, and the non-synchronicity shows up as a
|
||||||
|
strongly negative lag-1 autocorrelation and inflated variance — a portfolio risk figure built on it
|
||||||
|
would be overstated. It also has its own bad ticks (`NOK=X` falls 33 % on 2020-03-20 and rises 42 %
|
||||||
|
back on 2020-03-23; `EUR=X` moves −16 % then +14 % on 2008-12-08/09). Triangulation buys only the
|
||||||
|
removal of stale quotes (SEK 3.9 % of days unchanged, DKK 1.6 %), which costs far less than it adds.
|
||||||
|
|
||||||
|
## 2. Prices, FX and log-returns (`quantcore.data`, `quantcore.measurement.returns`)
|
||||||
|
|
||||||
|
* Adjusted closes `P_i(t)` in the listing currency on the union calendar of all exchanges, holidays
|
||||||
|
forward-filled (≤ 5 days; leading NaNs untouched).
|
||||||
|
* FX frame `X_c(t)` = CHF per unit of currency `c`, from Yahoo `cCHF=X`, crossed through USD when
|
||||||
|
the direct pair lacks history (`c/CHF = (USD/CHF)/(USD/c)`, e.g. PLN).
|
||||||
|
* Log-returns `r_i(t) = ln P_i(t) − ln P_i(t−1)`; CHF returns add the FX log-return:
|
||||||
|
`r_i^CHF(t) = r_i(t) + ln X_{c(i)}(t) − ln X_{c(i)}(t−1)`.
|
||||||
|
* Weekly returns: Friday-to-Friday closes; monthly: month-end to month-end.
|
||||||
|
* Horizon `h` in periods: day = 1, week = 5, month = 21 (daily data); at weekly frequency week = 1,
|
||||||
|
month = 4. Returns are assumed i.i.d., so `h`-period returns are sums.
|
||||||
|
* Portfolio log-return with weights `w` fixed at the start of the period (exact aggregation):
|
||||||
|
`r_p = ln(1 + Σ_i w_i (e^{r_i} − 1))`.
|
||||||
|
|
||||||
|
## 3. Estimators (`quantcore.measurement.estimators`)
|
||||||
|
|
||||||
|
Two families, both centred (the corresponding mean is subtracted):
|
||||||
|
|
||||||
|
* **Moving window** of `w` observations (default 200 days): sample mean, standard deviation
|
||||||
|
(ddof 1), covariance `np.cov`, correlation.
|
||||||
|
* **Exponentially weighted** with decay `λ` (defaults 0.94 for measurement, 0.99 for the slow
|
||||||
|
optimisation covariance): weight of the observation `k` periods back ∝ `λ^k`, normalised to one
|
||||||
|
(pandas `ewm(alpha=1−λ, adjust=True)`); covariance
|
||||||
|
`Σ = Σ_t ω_t (r_t − m)(r_t − m)'`, `m = Σ_t ω_t r_t`. Effective window ≈ `1/(1−λ)`.
|
||||||
|
* Streamed sequences `Σ_{t−1}` for every `t` (needed by the innovation filter): the rolling
|
||||||
|
estimator recomputes the window, the EWM estimator uses the exact centred recursion
|
||||||
|
`m_t = m_{t−1} + α d`, `Σ_t = λ Σ_{t−1} + α d (r_t − m_t)'`, `d = r_t − m_{t−1}`, `α = 1 − λ`.
|
||||||
|
* Linear shrinkage towards the diagonal `(1−s)Σ + s·diag(Σ)` with `s` either given or `"auto"`:
|
||||||
|
the Schäfer–Strimmer / Ledoit–Wolf intensity
|
||||||
|
`s* = Σ_{i≠j} Var̂(s_ij) / Σ_{i≠j} s_ij²` (clipped to [0, 1]) computed on the effective estimation
|
||||||
|
window (`w` observations, or `3/(1−λ)` for EWM). It is ≈ 0 for a long, strongly correlated
|
||||||
|
sample and → 1 when `N` approaches the number of observations — exactly when the sample
|
||||||
|
covariance becomes ill-conditioned. `"auto"` is the default of the optimisation covariance
|
||||||
|
(≈ 0.05 for 100 stocks / λ = 0.99, ≈ 0.11 for the full universe).
|
||||||
|
* Annualisation: means × periods per year (252 / 52 / 12), volatilities × √(periods per year).
|
||||||
|
|
||||||
|
## 4. Risk measures (`quantcore.measurement.risk`)
|
||||||
|
|
||||||
|
Losses are positive. For a log-return `r` over the horizon the loss is `−r` in return space and
|
||||||
|
`V₀(1 − e^{r})` in CHF (exact, monotone mapping — the CHF VaR/CVaR are obtained by applying it to
|
||||||
|
the return-space figures, or directly from money P&L for scenario sets).
|
||||||
|
|
||||||
|
* **Gaussian (GBM)**: with per-period mean `μ` and volatility `σ` from either estimator,
|
||||||
|
`μ_h = hμ`, `σ_h = √h σ`,
|
||||||
|
`VaR_β = −μ_h + z_β σ_h`, `CVaR_β = −μ_h + σ_h φ(z_β)/(1−β)`.
|
||||||
|
Reference: `σ = 1 %`, `μ = 0`: VaR₉₅ = 1.645 %, CVaR₉₅ = 2.063 %, VaR₉₉ = 2.326 %, CVaR₉₉ = 2.665 %.
|
||||||
|
* **Historical**: `VaR_β = −q_{1−β}(r_h)`, `CVaR_β = −E[r_h | r_h ≤ q_{1−β}]` on the last
|
||||||
|
`max(5w, 750)` observations aggregated to overlapping `h`-period sums.
|
||||||
|
* **Time series** (`risk_series`): rolling / EWM volatilities and the Gaussian VaR/CVaR built from
|
||||||
|
them, plus rolling historical quantiles and tail means.
|
||||||
|
* **Portfolio** (`portfolio_risk`): holdings in shares (+ cash) → value weights `w` at today's CHF
|
||||||
|
prices; Gaussian with `σ_p² = w'Σw`, historical on the realised portfolio return, or on a scenario
|
||||||
|
set (`method="scenarios"`).
|
||||||
|
* **Scenario CVaR** (`scenario_cvar`): with `S` equiprobable losses sorted descending and
|
||||||
|
`k = (1−β)S`, `CVaR_β = (Σ_{s≤⌊k⌋} L_(s) + (k−⌊k⌋) L_(⌊k⌋+1)) / k` — exactly what the
|
||||||
|
Rockafellar–Uryasev LP reproduces; `VaR_β = L_(⌈k⌉)`.
|
||||||
|
|
||||||
|
## 5. Performance (`quantcore.measurement.performance`)
|
||||||
|
|
||||||
|
Cumulative return `e^{Σr} − 1`; annualised log-return `ppy · mean(r)` and its geometric version
|
||||||
|
`e^{ppy·mean} − 1`; annualised volatility; Sharpe `(ann. return − r_f)/ann. vol` with the CHF
|
||||||
|
risk-free rate (`QUANTCORE_RISK_FREE_RATE`, default 0); Sortino (downside deviation); drawdown of
|
||||||
|
the cumulative value path, max drawdown, Calmar; skewness, excess kurtosis, hit ratio, best/worst
|
||||||
|
period. Rolling (window) and EWM (λ) versions as time series. Portfolios in shares: constant-mix
|
||||||
|
(today's value weights) or buy-and-hold.
|
||||||
|
|
||||||
|
## 6. Stock-price models (`quantcore.models.simulation`)
|
||||||
|
|
||||||
|
Both assume i.i.d. one-period log-returns; `Σ_T` is the latest matrix of the chosen
|
||||||
|
`CovarianceModel` (e.g. 200-day equally weighted, or EWM λ = 0.99).
|
||||||
|
|
||||||
|
* **Geometric Brownian motion**: `r_h ~ N(hm, hΣ_T)`, sampled with `A z`, `A A' = Σ_T`
|
||||||
|
(eigen-root with eigenvalues descending by default, so the first draw coordinate drives the market
|
||||||
|
factor; symmetric and Cholesky roots optional).
|
||||||
|
* **Empirical innovations** (filtered historical simulation):
|
||||||
|
`ε_t = Σ_{t−1}^{−1/2} r_t` with `Σ_{t−1}` estimated from data up to `t−1` (streamed sequence,
|
||||||
|
symmetric root `V Λ^{−1/2} V'` — Cholesky optional for large `N`); forward simulation
|
||||||
|
`r(j | T) = hm + Σ_T^{1/2} Σ_{k=1..h} ε_{j_k}` with the dates `j_k` drawn with replacement from
|
||||||
|
the history (scrambled Sobol' points on the market-factor ranks by default, `"mc"` uniform or
|
||||||
|
`"balanced"` optional). Innovations are centred so the scenario mean equals the supplied drift.
|
||||||
|
**Re-standardisation**: a lagged volatility filter leaves the innovations with variance above
|
||||||
|
one (the forecast is too low right after every shock — 1.1 for 10 stocks, 1.5 for 100 with
|
||||||
|
λ = 0.99), so the centred sample is rescaled before resampling: `standardize="diagonal"`
|
||||||
|
(default) divides each asset's innovations by their standard deviation, `"full"` whitens with
|
||||||
|
the innovations' sample covariance (needs `T ≥ 3N`). The scenario covariance then reproduces
|
||||||
|
`hΣ_T` (per-asset volatility within ±2 %, exactly with full whitening) while skewness, fat
|
||||||
|
tails and non-linear co-movements of the data are preserved. Innovations beyond ±8 standard
|
||||||
|
deviations are winsorised (a numerical guard affecting ≈ 0.2 % of the values); ill-conditioned
|
||||||
|
`Σ_{t−1}` for large `N` is handled by the automatic shrinkage of Section 3.
|
||||||
|
* **Sampling and moment matching** (`quantcore.models.sampling`): the optimiser is a sample-average
|
||||||
|
approximation and the sample mean of a plain Monte-Carlo set is off by `σ/√S` per asset — 3–13 %
|
||||||
|
p.a. for `S = 2000` — which the optimiser maximises (measured: two seeds give portfolios differing
|
||||||
|
by 100 % of wealth). Therefore: scrambled Sobol' points (`S` rounded up to a power of two),
|
||||||
|
**moment matching** `z̃ = C^{-1/2}(z − z̄)` so that the (probability-weighted) scenario mean and
|
||||||
|
covariance are *exactly* `hm` and `hΣ_T`, and the model expected returns in the optimiser's
|
||||||
|
objective. Only the shape of the tail remains sampled.
|
||||||
|
* **Drift convention**: the expected-return model delivers *arithmetic* expected returns `μ`
|
||||||
|
(per period, annual / 252). The log-drift is the Itô-corrected `m_i = ln(1 + hμ_i)/h − σ_i²/2`,
|
||||||
|
so that `E[e^{r_h} − 1] = hμ` exactly (`drift_convention="arithmetic"`); `"log"` uses `μ` as the
|
||||||
|
log-drift and adds `σ²/2` to the expected simple return (5 % p.a. for a typical stock, 34 % for
|
||||||
|
the most volatile one — an unintended tilt to volatile stocks).
|
||||||
|
* **Importance sampling** (`importance_sampling="auto"` = `"market"`): the loss direction `d` of
|
||||||
|
the *cap-weighted candidate portfolio*; two blocks of 25 % of the scenarios are shifted along it
|
||||||
|
by `z_0.95` and `z_0.99` (GBM: `z ↦ z + c_k d`; empirical: dates drawn with exponentially tilted
|
||||||
|
probabilities `q_t ∝ exp(ϑ_k d'ε_t)`), so one set serves both confidence levels. Every scenario
|
||||||
|
carries the self-normalised likelihood ratio of the defensive mixture (bounded by
|
||||||
|
`1/(1 − Σ mixture)`) as its probability `p_s`. Tail noise falls 2–4×; 2 048 tilted scenarios ≈
|
||||||
|
8 192 plain ones at β = 0.99. The shift is driven by the *covariance*: in return space it is
|
||||||
|
`−c_k Σv/√(v'Σv)`, i.e. each asset moves by its covariance with the reference portfolio (beta),
|
||||||
|
not by its own volatility — on the top 30 the correlation of the shift with beta is −1.00 versus
|
||||||
|
−0.93 with own volatility, and a negatively correlated stock is shifted *up*. The weighted
|
||||||
|
tilted set still reproduces `hΣ` and all correlations exactly (the unweighted one does not). The
|
||||||
|
gain a given portfolio gets scales with the correlation between its loss and the reference's
|
||||||
|
(`tilt_alignment` in the result: 5.3× at 1.00, 2.3× at 0.90 — the usual value for the optimiser's
|
||||||
|
solutions — and nothing below ~0.3, where a warning is raised). `"adaptive"` tilts towards a first-pass solution (sharper at 99 %,
|
||||||
|
but the set then depends on the risk parameters); `"off"` disables the tilt.
|
||||||
|
* **One scenario set per family of runs**: the scenario *key* (`OptimizationService.scenario_key`,
|
||||||
|
reported with every result) contains only model inputs — candidates, as-of date, frequency and
|
||||||
|
horizon, covariance model, scenario model, `n_scenarios`, sampling, moment matching, seed, tilt
|
||||||
|
mode, innovation settings. The CVaR limit, confidence, objective, constraints, costs, cash,
|
||||||
|
holdings and the expected-return model are *not* in it: varying them reuses the same scenarios
|
||||||
|
(cached in the service), so differences between runs are the effect of the parameter and not
|
||||||
|
sampling noise. The drift is applied after the random part (`ScenarioSet.with_drift`), so a new
|
||||||
|
view shifts the scenarios without touching the shocks (`shock_hash` is unchanged).
|
||||||
|
* `ScenarioSet` reports `expected_simple` (the expectation the optimiser uses), `probabilities`,
|
||||||
|
the effective sample size and a SHA-256 fingerprint `hash` of the matrix.
|
||||||
|
|
||||||
|
## 7. Expected returns — Black–Litterman with Idzorek confidence (`quantcore.models.black_litterman`)
|
||||||
|
|
||||||
|
Weekly CHF log-returns of the candidate stocks over the last 10 years (`bl_years`):
|
||||||
|
|
||||||
|
* `Σ` = 10-year sample covariance of weekly returns; `w` = current market-cap weights of the
|
||||||
|
candidates; `μ_M` = average weekly log-return of the **cap-weighted broad market index** of the
|
||||||
|
whole universe (weights drift with prices between rebalancings, today's shares held constant);
|
||||||
|
`r_f` = CHF risk-free rate per week.
|
||||||
|
* Risk aversion `δ = (μ_M − r_f)/(w'Σw)`; equilibrium excess returns `π = δ Σ w` (so the market
|
||||||
|
portfolio earns exactly `μ_M − r_f`).
|
||||||
|
* Views `P μ = Q + ε`, `ε ~ N(0, Ω)`, Ω diagonal. A view is `{assets: {ticker: weight}, q, confidence}`;
|
||||||
|
`q` is the annual total return of the view portfolio (converted to weekly excess internally;
|
||||||
|
the risk-free part cancels for relative views).
|
||||||
|
* **Idzorek confidence** `C_k ∈ (0, 1]`: the view's uncertainty is the `ω_k` for which the
|
||||||
|
unconstrained posterior weights move the fraction `C_k` of the way from `w` to the weights implied
|
||||||
|
by the view at 100 % confidence. For a diagonal Ω that calibration has the closed form
|
||||||
|
`ω_k = (1 − C_k)/C_k · p_k (τΣ) p_k'`; the tilt-matching minimisation of the paper is implemented
|
||||||
|
as `idzorek_omega_numeric` and the tests show both agree to 1e-4 (Idzorek 2005; the closed form is
|
||||||
|
due to Walters 2014).
|
||||||
|
* Posterior `μ_BL = π + τΣP'(PτΣP' + Ω)^{-1}(Q − Pπ)`,
|
||||||
|
`Σ_BL = Σ + τΣ − τΣP'(PτΣP' + Ω)^{-1}PτΣ`; `τ` default 0.05.
|
||||||
|
* Sentiment → view (`view_from_sentiment`): an assessment `s ∈ [−1, 1]` with degree of belief `C`
|
||||||
|
on stock `i` becomes the absolute view `q = π_i^{ann} + r_f + s · strength · σ_i^{ann}`,
|
||||||
|
`strength` = 0.25 by default (a fully positive view = +¼ annual volatility above equilibrium).
|
||||||
|
* Output: annual and per-period posterior total returns (used as scenario drift), Ω, δ, `μ_M`.
|
||||||
|
Alternatives: `expected_returns="historical"` (10-year means) or `"zero"`.
|
||||||
|
|
||||||
|
## 8. Mean–CVaR optimisation as a MILP (`quantcore.optimization.problem`)
|
||||||
|
|
||||||
|
Scenarios `R_{s,i}` = simple horizon returns `e^{r_{s,i}} − 1` in CHF (Section 6, with the drift of
|
||||||
|
Section 7), prices `P_i`, wealth `W = cash₀ + Σ_i n⁰_i P_i`.
|
||||||
|
|
||||||
|
Variables: shares `n_i ∈ ℤ₊` (no shorts); rebalancing adds buys `b_i`, sells `s_i ∈ ℤ₊` with
|
||||||
|
`n_i = n⁰_i + b_i − s_i`; `α`, `u_s ≥ 0` bounded by the worst gain/loss attainable within the share
|
||||||
|
caps (free variables make the CVaR objective numerically hard — cuOpt's concurrent method aborts on
|
||||||
|
such models; the client then retries once with its dual simplex); binaries `y_i` (cardinality / minimum position) and
|
||||||
|
`t_i` (fixed cost per trade) only when those options are used.
|
||||||
|
|
||||||
|
Transaction costs `TC = Σ_i (c_b P_i b_i + c_s P_i s_i) + F Σ_i t_i` (defaults 10 bp, no fixed cost).
|
||||||
|
|
||||||
|
Loss in scenario `s`: `L_s = TC − Σ_i P_i R_{s,i} n_i`. CVaR (Rockafellar–Uryasev):
|
||||||
|
|
||||||
|
```
|
||||||
|
u_s ≥ L_s − α (S rows)
|
||||||
|
CVaR_β = α + 1/(1−β) Σ_s p_s u_s (p_s = 1/S unless importance sampling is used)
|
||||||
|
```
|
||||||
|
|
||||||
|
Objectives: `max_return` — maximise `E[P&L] = Σ_i P_i μ_i n_i − TC` (model expected simple returns
|
||||||
|
`μ_i` over the horizon; the probability-weighted scenario mean if none are given) s.t. `CVaR_β ≤ L`
|
||||||
|
(the absolute CHF risk budget for the period, e.g. "CVaR₉₅ over a week ≤ 40 000"); `min_cvar`
|
||||||
|
s.t. `E[P&L] ≥ target`; `utility` — maximise `E[P&L] − λ·CVaR_β`.
|
||||||
|
|
||||||
|
Constraints: budget `cash₀ + Σ P_i s_i − Σ P_i b_i − TC ≥ 0` (cash may stay uninvested); single-stock
|
||||||
|
`P_i n_i ≤ w_max W` (10 %, implemented as tight variable bounds `n_i ≤ ⌊w_max W / P_i⌋`); sector
|
||||||
|
`Σ_{i∈k} P_i n_i ≤ w_k W` (default 25 %, per-sector dict possible); `TC ≤ TC_max`; turnover
|
||||||
|
`Σ P_i (b_i + s_i) ≤ τ W`; min/max invested fraction; cardinality `Σ y_i ≤ K`, `n_i ≤ M y_i`;
|
||||||
|
minimum position `P_i n_i ≥ v_min y_i`; `b_i + s_i ≤ M t_i`.
|
||||||
|
|
||||||
|
The reported expected P&L (from the model means), VaR and CVaR of the decoded integer portfolio are
|
||||||
|
recomputed from the weighted scenario set and agree with the solver's values (`tests/test_problem.py`,
|
||||||
|
including a brute-force optimality check). Note that `α` is only *a* minimiser of the RU function —
|
||||||
|
it equals the VaR only when the CVaR constraint is tight.
|
||||||
|
|
||||||
|
Sizes: `S + N(+2N) + 1` variables, `S + N + K + O(1)` rows, `≈ S·(N+2)` non-zeros; with the default
|
||||||
|
`N = 100`, `S = 4096` about 420 k non-zeros, solved by cuOpt in ~10 s. Solve times grow faster than
|
||||||
|
linearly in `S` (HiGHS LP ≈ `S^1.9`, cuOpt ≈ `S^1.2–1.6`): 2 000 → 100 000 scenarios costs ≈ 45× on
|
||||||
|
cuOpt.
|
||||||
|
|
||||||
|
**Tail-row generation** (`quantcore.optimization.rowgen`, `row_generation="auto"` from 8 192
|
||||||
|
scenarios on): only tail scenarios have `u_s > 0` at the optimum, so the MILP is solved on a working
|
||||||
|
set (the `2(1−β)S` worst scenarios of a reference portfolio), the losses of all scenarios are
|
||||||
|
checked, violated ones (`L_s > α*`) are added, and the loop stops when none is violated — the
|
||||||
|
restricted problem is a relaxation, so the final solution is optimal for the full set (within the
|
||||||
|
MIP gap). Measured: working sets ≈ 11 % of `S`, 2–3 iterations, 6–7× faster at 32 k–100 k scenarios.
|
||||||
|
|
||||||
|
**Exact Gaussian mode** (`scenario_model="gaussian_exact"`, `quantcore.optimization.gaussian_exact`):
|
||||||
|
with normal horizon returns `R ~ N(m, C)` the CVaR is `TC − m'v + k_β √(v'Cv)` (`v = P∘n`,
|
||||||
|
`k_β = φ(z_β)/(1−β)`); the mean–CVaR problem then coincides with the mean–variance problem (the
|
||||||
|
CVaR bound corresponds to an implicit variance bound `σ̄ = (Λ + m'v*)/k_β`). In integer shares this is
|
||||||
|
a convex MISOCP, solved by outer approximation: tangent cuts `t ≥ TC + (k_β C v_k/σ_k − m)'v` on the
|
||||||
|
same MILP with an epigraph variable `t` (`cvar`), a new cut at each solution until the true CVaR equals
|
||||||
|
`t` (20–70 cuts; identical to Gurobi's direct MISOCP solution to the cent). No scenarios, no sampling
|
||||||
|
error; the reference for the sampling tests.
|
||||||
|
|
||||||
|
## 9. Solvers (`quantcore.optimization.solvers`)
|
||||||
|
|
||||||
|
Both solvers receive the *same* `MILPModel` (objective, CSR matrix, row and variable bounds,
|
||||||
|
variable types); `solve_with_fallback` picks between them and the result always names the one that
|
||||||
|
produced it.
|
||||||
|
|
||||||
|
**cuOpt** — NVIDIA's GPU optimisation library, used through its self-hosted REST server
|
||||||
|
(`nvidia/cuopt`, version 26.08 here) running as the `cuopt` container on the DGX Spark:
|
||||||
|
`POST /cuopt/request` with a msgpack body, then poll `GET /cuopt/solution/{id}`; `"inf"/"ninf"` for
|
||||||
|
infinite bounds, `variable_types`, a mandatory `time_limit`. Statuses `Optimal` / `FeasibleFound`
|
||||||
|
(time limit — check `mip_gap`) / `Infeasible`. Docs: <https://docs.nvidia.com/cuopt/>. It is the
|
||||||
|
only part of the pipeline that uses the GPU.
|
||||||
|
|
||||||
|
**HiGHS** — the open-source LP/MILP/QP solver of ERGO-Code, University of Edinburgh (MIT licence,
|
||||||
|
<https://highs.dev>, source <https://github.com/ERGO-Code/HiGHS>, reference: Huangfu & Hall,
|
||||||
|
*Parallelizing the dual revised simplex method*, Math. Prog. Comp. 10, 119–142, 2018). quantcore
|
||||||
|
neither installs nor starts it separately: **HiGHS ships compiled inside SciPy**, and
|
||||||
|
`scipy.optimize.milp()` is a thin wrapper around it. It therefore runs **in-process, on the CPU, in
|
||||||
|
whichever process calls quantcore** — no network, no daemon, no separate container — and its version
|
||||||
|
follows the installed SciPy:
|
||||||
|
|
||||||
|
| where | Python | SciPy | HiGHS | notes |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `quantcore` container on the Spark (production) | 3.12.14 | 1.18.1 | 1.12.0 (`4f96ee8`) | aarch64 Linux, 4 threads (`OMP/OPENBLAS/MKL_NUM_THREADS=4`); numpy 2.5.3, pandas 3.0.6 |
|
||||||
|
| developer machine used for this work | 3.11.11 | 1.13.1 | 1.2.0 | arm64 macOS; numpy 1.26.4, pandas 2.1.4 |
|
||||||
|
|
||||||
|
The container tracks the newest releases (nothing is pinned beyond the lower bounds in
|
||||||
|
`pyproject.toml`), so a rebuild can move it a long way from the developer machine — the build of
|
||||||
|
2026-09-23 jumped it to pandas 3, where the month-end resampling alias is `'ME'` and the old `'M'`
|
||||||
|
is gone (`quantcore.measurement.returns.MONTH_END` resolves this at import time, and is the reason
|
||||||
|
monthly frequency works on both).
|
||||||
|
|
||||||
|
Verify it in any environment with
|
||||||
|
`python -c "import scipy.optimize as so, numpy as np, scipy.sparse as sp; print(so.milp(np.array([-3.,-2.]), constraints=so.LinearConstraint(sp.csr_matrix([[2.,3.],[4.,1.]]), [-np.inf]*2, [12.,10.]), integrality=np.ones(2), bounds=so.Bounds([0,0],[np.inf]*2), options={'disp': True}))"`,
|
||||||
|
which prints the HiGHS banner with its version.
|
||||||
|
|
||||||
|
HiGHS is used in four places: `solver="highs"`; as the fallback when cuOpt is unreachable or errors
|
||||||
|
(`solver="auto"`); for the cut MILPs of the exact Gaussian mode, which are small and numerous; and as
|
||||||
|
the oracle in the tests (brute-force optimality checks).
|
||||||
|
|
||||||
|
**Scenario count and candidate count**: `n_scenarios` defaults to 16 per candidate within
|
||||||
|
[4 096, 16 384]. The reported figures (expected P&L, CVaR) are stable at any size from ~2 000 on
|
||||||
|
(sd < 0.1 %), but the *composition* moves between near-equivalent portfolios in proportion to N/S —
|
||||||
|
3.5 % of wealth between seeds at N = 30, S = 2 048 against 12.5 % at N = 100 — and ~64 per candidate
|
||||||
|
is needed to hold it near 5 %. The result warns below 8 per candidate.
|
||||||
|
|
||||||
|
**Covariance rank**: an estimator with `T_eff` effective observations (rolling: the window; EWM:
|
||||||
|
`3/(1−λ)`) cannot support more than `T_eff` assets. A 200-day window with N = 300 gives 101 zero
|
||||||
|
eigenvalues and a portfolio with *negative* estimated variance; `shrinkage="auto"` (the default)
|
||||||
|
raises the intensity with N/T (0.033 at N = 30 → 0.128 at N = 745) and keeps the smallest eigenvalue
|
||||||
|
above 2 % of the average variance in every case. `CovarianceModel.conditioning()` reports the
|
||||||
|
figures and the optimisation warns when the estimate is singular or ill-conditioned for the size of
|
||||||
|
the candidate set. Long-only with a 10 % cap limits the damage in any case: the optimised portfolio's
|
||||||
|
out-of-sample/in-sample volatility ratio exceeds the cap-weighted portfolio's by only 0.89–1.18×
|
||||||
|
across N = 30…745, with or without shrinkage.
|
||||||
|
|
||||||
|
**cuOpt transport**: the model travels as one msgpack body (9.9 M non-zeros = 108 MB at N = 300,
|
||||||
|
S = 32 768), accepted at ~10 MB/s, so transport is not the constraint. The client uses a payload-scaled
|
||||||
|
timeout and a *fresh connection per attempt* — retrying on a pooled connection appends a second body and
|
||||||
|
the server rejects it with "unpack received extra data" (422) — and `solver="auto"` falls back to HiGHS
|
||||||
|
on failure. cuOpt's concurrent method also aborts on some CVaR-objective models; the client retries once
|
||||||
|
with its dual simplex (`method=2`). With row generation only the working set is sent.
|
||||||
|
|
||||||
|
**Which models go to cuOpt.** cuOpt 26.08 aborts on one specific combination: the CVaR in the
|
||||||
|
*objective* (`objective="min_cvar"`) **and** probability-weighted scenarios. The cause is the scaling
|
||||||
|
of the objective vector. In the min-CVaR formulation the coefficient on each excess `u_s` is
|
||||||
|
`p_s/(1−β)`, so the scenario probabilities *are* the objective coefficients; importance sampling
|
||||||
|
spreads them over about six orders of magnitude. Measured on a 4 137-variable / 4 106-row model
|
||||||
|
(40 candidates, 4 096 scenarios, β = 99 %):
|
||||||
|
|
||||||
|
| scenarios | objective `|c|` range | ratio | cuOpt |
|
||||||
|
|---|---|---|---|
|
||||||
|
| importance-sampled | 1.1e-06 … 1.0 | 8.8e5 | aborts in ~5 s |
|
||||||
|
| uniform (`importance_sampling="off"`) | 2.4e-02 … 1.0 | 4.1e1 | **Optimal in 3.8 s**, 5 runs out of 5, matching HiGHS to 7e-5 |
|
||||||
|
|
||||||
|
Relaxing the integrality does not help — the pure LP aborts too, so it is the LP and not the branch
|
||||||
|
and bound — and multiplying the objective by a constant does not either, since that leaves the ratio
|
||||||
|
unchanged. In the `max_return` formulation the same probabilities sit in a *constraint row* instead
|
||||||
|
(the objective is then just 40 expected returns, range 1e3), where cuOpt's row scaling copes, and it
|
||||||
|
solves at either weighting. Its methods are not interchangeable either: `method=1` (PDLP) returned the
|
||||||
|
weighted optimum once in 4.9 s and then aborted three times on the identical model, while `method=2`
|
||||||
|
and `method=3` always abort.
|
||||||
|
|
||||||
|
So `solver="auto"` diverts exactly that combination to HiGHS (1.7 s for that model) and the result
|
||||||
|
says why — otherwise a green cuOpt health pill next to `solver: "highs"` looks like the GPU being
|
||||||
|
ignored. Everything else keeps the GPU path. An explicit `solver="cuopt"` is always honoured and
|
||||||
|
returns cuOpt's own answer, abort included; an abort under `"auto"` is retried down
|
||||||
|
`CUOPT_RETRY_METHODS = (1, 2)` before the fallback.
|
||||||
|
|
||||||
|
**cuOpt vs HiGHS, both on the Spark** (end-to-end through the API: scenario generation, model build,
|
||||||
|
solve, decode; GBM model, weekly, β = 0.95, no tilt, 600 s solver time limit):
|
||||||
|
|
||||||
|
| case | cuOpt (GPU) | HiGHS (CPU, 4 threads) |
|
||||||
|
|---|---|---|
|
||||||
|
| N = 30, S = 8 192 | 15.6 s | 14.5 s |
|
||||||
|
| N = 30, S = 32 768 | 171.4 s | 171.0 s |
|
||||||
|
| N = 100, S = 4 096 | 10.0 s | 8.9 s |
|
||||||
|
| N = 100, S = 32 768 | 606.9 s (time limit, `FeasibleFound`) | 350.7 s (`Optimal`) |
|
||||||
|
| N = 300, S = 8 192 (row generation) | 139 s | 64 s |
|
||||||
|
|
||||||
|
HiGHS is at least as fast in every case measured and 2× faster on the two largest. That is not a
|
||||||
|
statement about GPUs in general: cuOpt's published speedups are for *LP* with its PDLP first-order
|
||||||
|
method, while [its own documentation](https://docs.nvidia.com/cuopt/user-guide/latest/milp-features.html)
|
||||||
|
states that for MILP the GPU runs primal heuristics and **branch and bound runs on the CPU**, that the
|
||||||
|
MIP solver is beta, and that "proving feasible solutions optimal remains under active development".
|
||||||
|
Our models have an easy LP relaxation and a small integrality gap, so the work sits almost entirely in
|
||||||
|
proving optimality — the part cuOpt does not accelerate — while HiGHS' dual simplex warm-starts across
|
||||||
|
branch-and-bound nodes. HiGHS is also a strong reference point: in the [Mittelmann MILP benchmark](https://plato.asu.edu/ftp/milp.html)
|
||||||
|
(7 July 2026) HiGHS 1.15.1 solves 179 of 240 instances at a scaled shifted geometric mean of 5.44,
|
||||||
|
against 1.72 for COPT 8.0.3 and 9.93 for SCIP. Caveats: cuOpt's `time_limit` covers only its internal
|
||||||
|
solve, it ignores the MIP-gap settings sent over REST (so it may prove optimality to a tighter
|
||||||
|
tolerance), and this is one problem family on one machine, not a benchmark.
|
||||||
|
|
||||||
|
**Reproducibility**: the scenario set depends only on the seed (`scenario_hash` in every result), and
|
||||||
|
the hash is identical for two runs **in the same environment**. Across environments it is not, and the
|
||||||
|
reason is the fingerprint rather than the mathematics. Measured between the developer Mac (numpy 1.26.4,
|
||||||
|
pandas 2.1.4, scipy 1.13.1) and the Spark container (numpy 2.5.3, pandas 3.0.6, scipy 1.18.1) on the
|
||||||
|
same 100 stocks and the same stored prices: scrambled Sobol' points and the numpy RNG stream are
|
||||||
|
bit-identical, the estimated covariance agrees to `max |dC| = 1.7e-18` (entries ~5e-4), its
|
||||||
|
eigenvalues to 1.4e-17 and the columns of `pca_root` to 4e-13 relative — no sign flips, no rotation.
|
||||||
|
The scenario matrices therefore agree to 13–14 significant digits. But `scenario_hash` fingerprints
|
||||||
|
`np.round(scenarios, 12)`, and on a 4096 × 100 matrix of weekly log-returns a *relative* perturbation
|
||||||
|
of 1e-15 already pushes ~3 entries of 409 600 across a rounding boundary (1e-14 pushes ~47), which
|
||||||
|
changes the digest. The figures are unaffected: the same request optimised on both machines gave
|
||||||
|
E[P&L] 1 888.65 against 1 888.76 and CVaR 39 999.71 against 39 999.82 on a CHF 1 M portfolio.
|
||||||
|
|
||||||
|
The portable identity is the **scenario key**, not the hash: it is built from the request and the
|
||||||
|
model description, so two environments that agree on the inputs produce the same `key` (verified:
|
||||||
|
`7162939ddf75ff74` on both machines for the run above) even where the digest of the realised matrix
|
||||||
|
differs. Read `scenario_hash` as "these two results used bit-identical scenarios" — a statement about
|
||||||
|
one machine — and `scenario_key` as "these two results were asked for the same scenarios".
|
||||||
|
HiGHS' branch-and-bound is deterministic → identical shares for the same inputs *and the same HiGHS
|
||||||
|
build* (`reproducibility="exact"`); across SciPy versions (1.2.0 on the developer machine against
|
||||||
|
1.12.0 in the container) the search can end on a different optimal vertex among ties, with the same
|
||||||
|
objective. cuOpt 26.08
|
||||||
|
ignores the MIP-gap settings sent over REST and stops within its default relative gap of 1e-4: the
|
||||||
|
objective, expected P&L and CVaR of two runs agree to that gap (≈ 0.2 CHF/week on CHF 1 M) while
|
||||||
|
near-tied share counts can differ by a few shares (`reproducibility="within_gap"`). The Gaussian exact
|
||||||
|
mode runs on HiGHS by default.
|
||||||
|
|
||||||
|
## 10. Default parameters
|
||||||
|
|
||||||
|
| Parameter | Default | Where |
|
||||||
|
|---|---|---|
|
||||||
|
| history start / required first price | 2004-12-01 / ≤ 2005-01-31, coverage ≥ 90 % | `Settings` |
|
||||||
|
| base currency, risk-free | CHF, 0 % | `Settings` |
|
||||||
|
| measurement window / λ | 200 periods / 0.94 | `RiskService` |
|
||||||
|
| optimisation covariance | EWM λ = 0.99 (≈ 100-day effective window) | `OptimizationRequest.covariance` |
|
||||||
|
| scenarios | empirical innovations, 4 096 scrambled Sobol' draws on the market-factor ranks, moment matched, arithmetic drift, last 2 500 days, diagonal re-standardisation, seed 42 | `OptimizationRequest` |
|
||||||
|
| importance sampling / row generation | `auto` = market tilt (blocks at z₀.₉₅, z₀.₉₉, 25 % each); row generation from 8 192 scenarios | `OptimizationRequest` |
|
||||||
|
| MIP gap | 1e-4 relative (HiGHS; cuOpt uses its own default) | `Settings.mip_rel_gap` |
|
||||||
|
| Black–Litterman | 10 years weekly, τ = 0.05, sentiment strength 0.25 | `OptimizationRequest`, `view_from_sentiment` |
|
||||||
|
| constraints | 10 % per stock, 25 % per sector, no shorts, 10 bp costs | `OptimizationConstraints`, `TransactionCosts` |
|
||||||
|
| CVaR limit | explicit CHF amount or fraction (5 % of wealth if nothing is given) | `OptimizationRequest` |
|
||||||
|
| cuOpt time limit | 120 s (`QUANTCORE_CUOPT_TIME_LIMIT`) | `Settings` |
|
||||||
254
docs/SPARK_APPTAINER.md
Normal file
254
docs/SPARK_APPTAINER.md
Normal file
@ -0,0 +1,254 @@
|
|||||||
|
# quantcore on your DGX Spark (Apptainer, no root)
|
||||||
|
|
||||||
|
This guide sets up the quant service of the project on your own DGX Spark. The service covers
|
||||||
|
market data, risk measurement, Black–Litterman expected returns and the mean-CVaR optimiser.
|
||||||
|
Everything runs on the Spark itself:
|
||||||
|
|
||||||
|
- **cuOpt**, NVIDIA's GPU solver, runs in its own Apptainer container.
|
||||||
|
- **quantcore**, this repository, runs in a second Apptainer container. It calls cuOpt at
|
||||||
|
`http://127.0.0.1:5000`.
|
||||||
|
- The API and the browser test bench listen on `http://127.0.0.1:5001`. You reach them from the
|
||||||
|
Spark: its desktop browser, a terminal, a notebook, or the platform you build. They are not
|
||||||
|
reachable from the network.
|
||||||
|
|
||||||
|
You need no root rights, no Docker and no VPN. One script, `deploy/apptainer/qc.sh`, does
|
||||||
|
everything.
|
||||||
|
|
||||||
|
## What you need
|
||||||
|
|
||||||
|
| Requirement | Check |
|
||||||
|
|---|---|
|
||||||
|
| Apptainer 1.1 or newer | `apptainer --version` |
|
||||||
|
| The GPU | `nvidia-smi -L` lists the NVIDIA GB10. Without it, optimisation runs on HiGHS on the CPU. |
|
||||||
|
| About 5 GB in your home directory | The cuOpt image is 3.3 GB, quantcore 0.2 GB, the data 40 MB. Pulling cuOpt needs about 12 GB for a few minutes. |
|
||||||
|
| Internet access | Docker Hub for the images, PyPI for the Python libraries, Yahoo Finance for market data. |
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management.git
|
||||||
|
cd Portfolio_Risk_Management
|
||||||
|
|
||||||
|
deploy/apptainer/qc.sh install # once, ~10 min: settings, cuOpt image, quantcore image, self-test
|
||||||
|
deploy/apptainer/qc.sh data-import /path/to/quantcore_data.tar.gz # the prepared data set
|
||||||
|
# or build the data set from Yahoo Finance yourself (~15 min): deploy/apptainer/qc.sh setup
|
||||||
|
deploy/apptainer/qc.sh start # starts cuOpt and the API
|
||||||
|
```
|
||||||
|
|
||||||
|
Then open **http://127.0.0.1:5001/ui/** in the Spark's browser. The interactive API
|
||||||
|
documentation is at **http://127.0.0.1:5001/docs**.
|
||||||
|
|
||||||
|
Working on the Spark over SSH from your laptop? Forward the port, then open the same address in
|
||||||
|
the laptop's browser. Nothing else on the network can reach the service:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -L 5001:127.0.0.1:5001 <your-user>@<your-spark>
|
||||||
|
```
|
||||||
|
|
||||||
|
A shorter command saves typing. Put this line in your `~/.bashrc`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
alias qc="$HOME/Portfolio_Risk_Management/deploy/apptainer/qc.sh"
|
||||||
|
```
|
||||||
|
|
||||||
|
The rest of this guide writes `qc` for the script.
|
||||||
|
|
||||||
|
## Daily use
|
||||||
|
|
||||||
|
```bash
|
||||||
|
qc start # cuOpt + API; after a reboot, run it again
|
||||||
|
qc status # what runs, whether cuOpt is reachable, how much data there is
|
||||||
|
qc logs # API log; `qc logs cuopt` for the solver, add -f to follow
|
||||||
|
qc update # the newest prices and FX rates from Yahoo Finance
|
||||||
|
qc stop # stop both containers
|
||||||
|
```
|
||||||
|
|
||||||
|
The containers keep running after you log out. They stop when the Spark reboots.
|
||||||
|
|
||||||
|
## Using the service
|
||||||
|
|
||||||
|
**In the browser.** The test bench at `http://127.0.0.1:5001/ui/` shows every service: the
|
||||||
|
stock universe, risk and scenarios, expected returns and views, constraints, and the
|
||||||
|
optimisation with a before/after comparison.
|
||||||
|
|
||||||
|
**From your own code.** This is how the platform you build uses the service. The API speaks
|
||||||
|
JSON over HTTP, so any language works. Your code reaches it at `http://127.0.0.1:5001`. That
|
||||||
|
address also works inside your own Apptainer containers, because Apptainer containers share
|
||||||
|
the Spark's network. For Python there is a ready-made client:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from quantcore.client import QuantcoreClient
|
||||||
|
|
||||||
|
qc = QuantcoreClient("http://127.0.0.1:5001")
|
||||||
|
print(qc.health()["cuopt"]["reachable"])
|
||||||
|
risk = qc.asset_risk(["NESN.SW", "AAPL"], horizon="week")
|
||||||
|
```
|
||||||
|
|
||||||
|
Install the client into your own environment with
|
||||||
|
`pip install "git+https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management.git"`.
|
||||||
|
[USAGE.md](USAGE.md) describes every endpoint and the request fields.
|
||||||
|
|
||||||
|
**From the command line, through the API.** `qc api` runs the API client against your
|
||||||
|
service:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
qc api health
|
||||||
|
qc api selftest # calls every endpoint, pass/fail table
|
||||||
|
qc api risk NESN.SW AAPL --horizon week
|
||||||
|
qc api optimize --cash 1000000 --cvar-limit 40000 --horizon week
|
||||||
|
```
|
||||||
|
|
||||||
|
**From the command line, without the API.** `qc cli` runs the services directly in the
|
||||||
|
container. The API does not need to be running:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
qc cli status
|
||||||
|
qc cli risk NESN.SW AAPL --horizon week
|
||||||
|
qc cli optimize --cash 1000000 --cvar-limit 40000 --horizon week --max-assets 50
|
||||||
|
```
|
||||||
|
|
||||||
|
## Settings
|
||||||
|
|
||||||
|
`qc install` writes the settings file `~/quantcore/quantcore.env`. Edit it, then run
|
||||||
|
`qc restart`. A value set in the shell wins over the file, for example
|
||||||
|
`QC_API_PORT=5011 qc start`.
|
||||||
|
|
||||||
|
| Setting | Default | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `QC_API_PORT` | `5001` | Port of the API and the test bench. |
|
||||||
|
| `QC_CUOPT_PORT` | `5000` | Port of the cuOpt server. |
|
||||||
|
| `QC_CUOPT_START` | `auto` | `auto` uses a cuOpt that already answers on the port, or starts one. `yes` always starts its own. `no` never starts one, so HiGHS solves every model. |
|
||||||
|
| `QC_CUOPT_IMAGE` | `docker://nvidia/cuopt:26.8.0-cuda13.3-py3.14` | Where the cuOpt image comes from. Set the path of a `.sif` file if your Spark already provides one. |
|
||||||
|
| `QC_IMAGE` | `build` | How quantcore gets its container; see the next section. |
|
||||||
|
| `QUANTCORE_*` | | Passed into the container unchanged. Examples are `QUANTCORE_CUOPT_TIME_LIMIT` and `QUANTCORE_ENABLE_UI`. All of them are listed in `quantcore/config.py`. |
|
||||||
|
|
||||||
|
**Several users on one Spark.** Ports belong to the whole machine. If someone else already uses
|
||||||
|
port 5001, `qc start` stops and suggests a free port to put into your settings file. A cuOpt
|
||||||
|
that another user already runs on port 5000 is simply used: cuOpt keeps no state between
|
||||||
|
requests.
|
||||||
|
|
||||||
|
## How quantcore gets its container
|
||||||
|
|
||||||
|
`QC_IMAGE=build` is the default. `qc install` then builds the image `quantcore.sif` from this
|
||||||
|
repository with Apptainer's unprivileged build. Some Spark installations do not allow
|
||||||
|
unprivileged builds. In that case `qc install` falls back by itself to a setup that needs no
|
||||||
|
build. It pulls a plain Python image and installs quantcore into a virtual environment in
|
||||||
|
`~/quantcore/venv`. The libraries and the results are the same, because both setups install the
|
||||||
|
pinned versions in `deploy/apptainer/constraints.txt`. You can choose the fallback directly
|
||||||
|
with `QC_IMAGE=venv`.
|
||||||
|
|
||||||
|
`QC_IMAGE` also accepts the path of a prebuilt `quantcore.sif`, or a `docker://` or `oras://`
|
||||||
|
address to pull one from.
|
||||||
|
|
||||||
|
**After `git pull`**, rebuild and restart:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git pull && qc build && qc restart
|
||||||
|
```
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
|
||||||
|
```
|
||||||
|
DGX Spark, your user account, no root
|
||||||
|
┌────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ browser / your platform / qc api │
|
||||||
|
│ │ HTTP │
|
||||||
|
│ ▼ │
|
||||||
|
│ 127.0.0.1:5001 instance "quantcore-api" quantcore.sif │
|
||||||
|
│ FastAPI + test bench ~/quantcore/data -> /data │
|
||||||
|
│ │ HTTP (msgpack) │
|
||||||
|
│ ▼ │
|
||||||
|
│ 127.0.0.1:5000 instance "quantcore-cuopt" cuopt.sif, started with --nv │
|
||||||
|
│ cuOpt server ──► GB10 GPU │
|
||||||
|
└────────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
- An Apptainer **instance** is a container that runs in the background, like a service.
|
||||||
|
`apptainer instance list` shows yours.
|
||||||
|
- Apptainer containers use the Spark's network directly, with no port mapping. A server inside a
|
||||||
|
container that listens on `127.0.0.1` is therefore reachable from the Spark, and only from it.
|
||||||
|
- The images are read-only. Everything that changes lives in `~/quantcore/data`, which is
|
||||||
|
mounted into the container as `/data`.
|
||||||
|
- The containers start with `--cleanenv`. Variables from your shell, such as an activated
|
||||||
|
virtual environment or `PYTHONPATH`, cannot interfere. `qc.sh` passes only the settings above,
|
||||||
|
plus proxy variables if you have them.
|
||||||
|
- NVIDIA's cuOpt image sets its home directory to `/opt/cuopt`. That folder is writable under
|
||||||
|
Docker but read-only in an Apptainer image, and the solver would crash on its first job when
|
||||||
|
CuPy writes its kernel cache there. `qc.sh` therefore starts the cuOpt server with its home
|
||||||
|
directory in `~/quantcore/cuopt-home`.
|
||||||
|
- When cuOpt is not reachable, or stops during a solve, every optimisation falls back to HiGHS on
|
||||||
|
the CPU. The result says which solver produced it.
|
||||||
|
|
||||||
|
| Path | Content |
|
||||||
|
|---|---|
|
||||||
|
| `~/quantcore/quantcore.env` | your settings |
|
||||||
|
| `~/quantcore/data/` | universe, prices, FX, snapshots |
|
||||||
|
| `~/quantcore/images/cuopt.sif` | the cuOpt image |
|
||||||
|
| `~/quantcore/images/quantcore.sif` | the quantcore image; in the fallback, `python.sif` plus `~/quantcore/venv/` |
|
||||||
|
| `~/quantcore/cuopt-home/` | cuOpt's caches (CuPy kernels, CUDA) |
|
||||||
|
| `~/.apptainer/instances/logs/` | instance logs, also shown by `qc logs` |
|
||||||
|
| `~/.apptainer/cache/` | downloaded image layers; `qc clean-cache` frees the space |
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
| Symptom | Cause and fix |
|
||||||
|
|---|---|
|
||||||
|
| `port 5001 is taken` | Another user or program uses the port. Set the suggested `QC_API_PORT` in `~/quantcore/quantcore.env`, then run `qc start`. |
|
||||||
|
| `qc status` shows cuOpt as not answering | Run `qc logs cuopt`. Check that `nvidia-smi` works. Then `qc restart`. Optimisations keep working on HiGHS in the meantime. |
|
||||||
|
| The cuOpt log says `Read-only file system: '/opt/cuopt/.cupy'` | cuOpt was started without `qc.sh`. Start it with `qc start`, or give it a writable home directory as described in "Running cuOpt by hand". |
|
||||||
|
| `/optimize` answers 503, "no market data yet" | Run `qc data-import <file>` or `qc setup`. |
|
||||||
|
| Yahoo errors during `qc setup` or `qc update` | Yahoo limits how fast it answers. Run the command again later; stored data stays intact. |
|
||||||
|
| `qc install` stops with "no 'instance run'" | The Spark's Apptainer is older than 1.1. Ask the administrator to update it. |
|
||||||
|
| The cuOpt pull fails with "no space left on device" | Apptainer unpacks images in `/tmp`. Point it at a larger directory: `APPTAINER_TMPDIR=~/quantcore/tmp qc pull-cuopt`. |
|
||||||
|
| Model status `FeasibleFound` instead of `Optimal` | The time limit was reached. Raise `time_limit` in the request or `QUANTCORE_CUOPT_TIME_LIMIT`, or reduce `n_scenarios` or `max_assets`. |
|
||||||
|
|
||||||
|
## Running cuOpt by hand
|
||||||
|
|
||||||
|
`qc start` does this for you. If you want to run the cuOpt image yourself, for example for another
|
||||||
|
project, start it like this:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
apptainer pull cuopt.sif docker://nvidia/cuopt:26.8.0-cuda13.3-py3.14
|
||||||
|
mkdir -p ~/cuopt-home
|
||||||
|
apptainer instance run --nv --cleanenv --bind ~/cuopt-home cuopt.sif my-cuopt \
|
||||||
|
env HOME=$HOME/cuopt-home python -m cuopt_server.cuopt_service -i 127.0.0.1 -p 5000
|
||||||
|
curl http://127.0.0.1:5000/cuopt/health
|
||||||
|
apptainer instance stop my-cuopt
|
||||||
|
```
|
||||||
|
|
||||||
|
## For the lecturer
|
||||||
|
|
||||||
|
**Data set for the students.** Many students running `qc setup` at the same time from one
|
||||||
|
network can hit Yahoo's rate limits. A prepared data set avoids that, and every student then
|
||||||
|
works on the same data. Pack it on a machine that has the data:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
qc data-export quantcore_data.tar.gz # or, from any data directory:
|
||||||
|
tar -czf quantcore_data.tar.gz -C <data dir> --exclude ./jobs --exclude '*.log' .
|
||||||
|
```
|
||||||
|
|
||||||
|
The archive is about 30 MB. It is not part of the public repository: Yahoo's terms do not allow
|
||||||
|
redistributing their data.
|
||||||
|
|
||||||
|
**Prebuilt image.** Build `quantcore.sif` on any aarch64 Linux machine where you can build
|
||||||
|
images, then hand out the file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
apptainer build quantcore.sif deploy/apptainer/quantcore.def # from the repository root
|
||||||
|
```
|
||||||
|
|
||||||
|
Students set `QC_IMAGE=/path/to/quantcore.sif` in their settings file. A copy built and tested on
|
||||||
|
the lecturer's Spark is described in the verification note below.
|
||||||
|
|
||||||
|
**The lecturer's own Spark** runs the Docker setup in [SPARK_DEPLOYMENT.md](SPARK_DEPLOYMENT.md).
|
||||||
|
That setup needs root and Docker. Students do not need it.
|
||||||
|
|
||||||
|
**Verification.** The setup was tested end to end on 7 October 2026 on a DGX Spark: NVIDIA GB10,
|
||||||
|
driver 580.159, Ubuntu 24.04, Apptainer 1.5.4, as a user without root. That Spark allowed no
|
||||||
|
unprivileged image builds, so `qc install` took the virtualenv fallback by itself. The image mode
|
||||||
|
was then tested with a prebuilt `quantcore.sif`. In both modes:
|
||||||
|
|
||||||
|
- `qc api selftest --full` passed on every endpoint, and the optimisation ran on cuOpt on the GPU.
|
||||||
|
- `qc update` fetched the newest Yahoo prices through the running API.
|
||||||
|
- A taken API port stopped `qc start` with a suggested free port, and that port worked.
|
||||||
|
- The test suite passed inside the container, including the round trip to the live cuOpt server.
|
||||||
110
docs/SPARK_DEPLOYMENT.md
Normal file
110
docs/SPARK_DEPLOYMENT.md
Normal file
@ -0,0 +1,110 @@
|
|||||||
|
# Running the quant service on the DGX Spark (Docker, with root)
|
||||||
|
|
||||||
|
> This is the setup of the lecturer's Spark, which has root rights, Docker Compose and Tailscale.
|
||||||
|
> **Students** run the service on their own Spark with Apptainer and without root:
|
||||||
|
> see [SPARK_APPTAINER.md](SPARK_APPTAINER.md).
|
||||||
|
|
||||||
|
The Spark (`spark-6d4a`, reachable over Tailscale) already runs cuOpt, Gurobi and JupyterLab as
|
||||||
|
Docker containers defined in `~/services/docker-compose.yml`. The quant service joins that
|
||||||
|
compose file as the container **`quantcore`** on port **5001**; it reaches cuOpt over the compose
|
||||||
|
network (`http://cuopt:5000`) and keeps its data on the host in `~/services/work/quantcore_data`.
|
||||||
|
|
||||||
|
| Service | Address | What it is |
|
||||||
|
|---|---|---|
|
||||||
|
| **quantcore** | `http://spark-6d4a:5001` (`/docs` for Swagger, `/ui/` for the test bench) | portfolio data, risk/performance measurement, models, mean-CVaR MILP |
|
||||||
|
| cuOpt | `http://spark-6d4a:5000` | the GPU MILP solver the service uses |
|
||||||
|
|
||||||
|
## Deploy (from the Mac)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
deploy/deploy_to_spark.sh --with-data # first time: also ships the locally built data set
|
||||||
|
deploy/deploy_to_spark.sh # later: code only
|
||||||
|
```
|
||||||
|
|
||||||
|
The script rsyncs the repository to `~/services/work/Portfolio_Risk_Optimization`, appends the
|
||||||
|
service block from `deploy/docker-compose.quantcore.yml` to the compose file (once; a backup of the
|
||||||
|
compose file is kept), builds the image (`python:3.12-slim`, aarch64) and starts the container.
|
||||||
|
|
||||||
|
The block is appended **only if `quantcore:` is not already in `~/services/docker-compose.yml`**, so
|
||||||
|
later edits to `deploy/docker-compose.quantcore.yml` (a new environment variable, a changed port) do
|
||||||
|
not travel with a redeploy — apply them to the Spark's compose file by hand, or delete the block
|
||||||
|
there and re-run the script. Code and image changes need no such step.
|
||||||
|
|
||||||
|
Without `--with-data` the container starts empty; run the setup once through the API:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://spark-6d4a:5001/jobs/setup # ~15 min: 1 117 tickers' metadata + history
|
||||||
|
curl http://spark-6d4a:5001/jobs/<id> # status: queued | running | done | failed
|
||||||
|
```
|
||||||
|
|
||||||
|
## Operate
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh user_fh_fhgr@spark-6d4a
|
||||||
|
cd ~/services
|
||||||
|
docker compose ps quantcore # status (healthcheck on /health)
|
||||||
|
docker compose logs -f quantcore # live logs
|
||||||
|
docker compose restart quantcore
|
||||||
|
docker compose up -d --build quantcore # rebuild after a code change (or re-run the deploy script)
|
||||||
|
|
||||||
|
# inside the container: the CLI works against the same data volume
|
||||||
|
docker exec -it quantcore quantcore status
|
||||||
|
docker exec -it quantcore quantcore update # newest prices (incremental)
|
||||||
|
docker exec -it quantcore quantcore snapshot --name demo
|
||||||
|
```
|
||||||
|
|
||||||
|
Nightly data refresh (host cron on the Spark), e.g. `crontab -e`:
|
||||||
|
|
||||||
|
```
|
||||||
|
15 23 * * 1-5 curl -s -X POST http://localhost:5001/jobs/update >/dev/null
|
||||||
|
```
|
||||||
|
|
||||||
|
## Use from anywhere on the tailnet
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl http://spark-6d4a:5001/health
|
||||||
|
curl "http://spark-6d4a:5001/universe?region=Europe§or=Health%20Care"
|
||||||
|
curl -s http://spark-6d4a:5001/risk/assets -H 'Content-Type: application/json' \
|
||||||
|
-d '{"symbols": ["NESN.SW", "ROP.SW", "AAPL"], "horizon": "week"}'
|
||||||
|
curl -s http://spark-6d4a:5001/optimize -H 'Content-Type: application/json' \
|
||||||
|
-d '{"cash": 1000000, "max_assets": 100, "cvar_limit": 40000, "horizon": "week"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
The test bench is served from the same container (`QUANTCORE_ENABLE_UI`, default `true`, in the
|
||||||
|
compose block). Set it to `"false"` and rebuild for a deployment that exposes the JSON API and
|
||||||
|
`/docs` only — for users who build their own front end against these endpoints.
|
||||||
|
`deploy/docker-compose.students.yml` is that deployment, standalone and with the bench already
|
||||||
|
off; `docs/USAGE.md` is the guide its users need.
|
||||||
|
|
||||||
|
Python clients simply point `QUANTCORE_CUOPT_URL` / the API base URL at the Spark; the package
|
||||||
|
itself also runs on a laptop (`quantcore serve`) and uses the Spark's cuOpt remotely
|
||||||
|
(`QUANTCORE_CUOPT_URL=http://spark-6d4a:5000`), falling back to HiGHS — which needs no service of its
|
||||||
|
own: it is compiled into SciPy inside the `quantcore` image (SciPy 1.18.1 → HiGHS 1.12.0) and runs on
|
||||||
|
the container's CPU — when the
|
||||||
|
Spark is unreachable.
|
||||||
|
|
||||||
|
## Sizing
|
||||||
|
|
||||||
|
| Candidates | Scenarios | MILP size (vars / rows / nnz) | cuOpt solve |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 100 | 2 000 | 2 101 / 2 013 / 206 k | ~6–10 s |
|
||||||
|
| 300 | 2 000 | 2 301 / 2 013 / 607 k | ~12 s |
|
||||||
|
| 745 (all) | 1 000 | 1 746 / 1 013 / 749 k | ~17 s |
|
||||||
|
| 100, rebalancing with fixed costs | 2 000 | 2 401 / 2 215 / 808 k (400 integers) | ~33 s |
|
||||||
|
|
||||||
|
The time limit (`QUANTCORE_CUOPT_TIME_LIMIT`, request field `time_limit`) bounds the solve; a
|
||||||
|
`FeasibleFound` status with a small `mip_gap` is a valid, slightly sub-optimal answer.
|
||||||
|
|
||||||
|
BLAS threads are pinned to 4 (`OPENBLAS_NUM_THREADS`): the empirical-innovation filter performs
|
||||||
|
thousands of small eigendecompositions, which OpenBLAS runs ~50× slower with all 20 cores.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
| Symptom | Cause / fix |
|
||||||
|
|---|---|
|
||||||
|
| `/health` shows `cuopt.reachable: false` | cuOpt container down or unhealthy: `docker compose restart cuopt` (see the Spark README §5). Optimisations fall back to HiGHS and say so in `solver` / `warnings`. |
|
||||||
|
| `/optimize` answers 503 "no market data yet" | run `POST /jobs/setup` or deploy with `--with-data`. |
|
||||||
|
| Yahoo errors in the setup/update job | rate limiting; the client retries with back-off and opens a circuit after 5 failures. Re-run the job later; stored data stays intact. |
|
||||||
|
| `FeasibleFound` instead of `Optimal` | time limit hit; raise `time_limit` or reduce `n_scenarios` / `max_assets`. |
|
||||||
|
| `Infeasible` | the constraints contradict each other (e.g. `min_invested` with a tiny CVaR limit, or a rebalancing turnover cap that prevents reaching the risk budget). |
|
||||||
|
| Tailscale SSH asks for a browser check | `ssh user_fh_fhgr@spark-6d4a` prints a login URL; open it once, then re-run the deploy script. |
|
||||||
342
docs/USAGE.md
Normal file
342
docs/USAGE.md
Normal file
@ -0,0 +1,342 @@
|
|||||||
|
# Using quantcore
|
||||||
|
|
||||||
|
`quantcore` is the **deterministic quant core** your advisory platform builds on: market data, risk
|
||||||
|
and performance measurement, stock-price models, Black–Litterman expected returns and mean–CVaR
|
||||||
|
portfolio optimisation in whole shares.
|
||||||
|
|
||||||
|
It contains **no LLM call and makes no decisions**. It computes numbers from data and from the inputs
|
||||||
|
you hand it, and every answer carries the provenance of those inputs. Your platform supplies the
|
||||||
|
judgement — which news matters, how strongly to believe it, what to tell the client — and calls in
|
||||||
|
here for every figure it quotes.
|
||||||
|
|
||||||
|
You never have to modify this package. Treat it as a service with a contract.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Two ways to use it
|
||||||
|
|
||||||
|
| | **HTTP API** | **Python library** |
|
||||||
|
|---|---|---|
|
||||||
|
| you need | a URL | the package *and* the data directory on the same machine |
|
||||||
|
| import | nothing — `requests`, or `quantcore.client` | `from quantcore.measurement.risk import RiskService` |
|
||||||
|
| language | any | Python |
|
||||||
|
| good for | the normal case: your platform calls a running service | notebooks, experiments, running your own backtest loop |
|
||||||
|
|
||||||
|
Start with the API. Everything in this guide is available both ways, and the library classes are
|
||||||
|
documented in `README.md` and in `docs/ARCHITECTURE.md` if you need them.
|
||||||
|
|
||||||
|
## 2. Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management.git
|
||||||
|
cd Portfolio_Risk_Management
|
||||||
|
pip install -e . # Python 3.11+
|
||||||
|
```
|
||||||
|
|
||||||
|
On a DGX Spark you do not need this step to run the service: `deploy/apptainer/qc.sh` runs it in
|
||||||
|
containers (see §3 and [SPARK_APPTAINER.md](SPARK_APPTAINER.md)). Install the package only where your
|
||||||
|
own code imports it, for example for `quantcore.client`.
|
||||||
|
|
||||||
|
This gives you two commands:
|
||||||
|
|
||||||
|
| command | what it is |
|
||||||
|
|---|---|
|
||||||
|
| `quantcore` | runs the services **in this process**; needs the data directory locally |
|
||||||
|
| `quantcore-api` | talks to a **running** service over HTTP; needs only its URL |
|
||||||
|
|
||||||
|
## 3. Point at a service, or run your own
|
||||||
|
|
||||||
|
**On your DGX Spark** (the usual case): the service and cuOpt run as Apptainer containers of your
|
||||||
|
user, reachable at `127.0.0.1` on the Spark only. [SPARK_APPTAINER.md](SPARK_APPTAINER.md) has the details:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
deploy/apptainer/qc.sh install && deploy/apptainer/qc.sh data-import quantcore_data.tar.gz
|
||||||
|
deploy/apptainer/qc.sh start # API at http://127.0.0.1:5001 (the client's default URL)
|
||||||
|
deploy/apptainer/qc.sh api health
|
||||||
|
```
|
||||||
|
|
||||||
|
**Someone else runs it:** point the client at that machine.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export QUANTCORE_API_URL=http://<host>:5001
|
||||||
|
quantcore-api health
|
||||||
|
```
|
||||||
|
|
||||||
|
**You run it yourself.** The data has to be built once, which downloads about 1 100 tickers from
|
||||||
|
Yahoo Finance and takes ~15 minutes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
quantcore setup # once
|
||||||
|
quantcore serve # HTTP API on :5001, interactive docs at /docs
|
||||||
|
```
|
||||||
|
|
||||||
|
Or as a container, which is how it runs in production:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker build -f deploy/Dockerfile -t quantcore:0.1 .
|
||||||
|
docker run -p 5001:5001 -v $PWD/data:/data \
|
||||||
|
-e QUANTCORE_CUOPT_URL=http://<your-cuopt-host>:5000 quantcore:0.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Configuration is environment variables with a `QUANTCORE_` prefix (see `.env.example`); the ones you
|
||||||
|
are likely to touch:
|
||||||
|
|
||||||
|
| variable | default | meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `QUANTCORE_API_URL` | `http://127.0.0.1:5001` | which service `quantcore-api` and `QuantcoreClient` talk to |
|
||||||
|
| `QUANTCORE_DATA_DIR` | `./data` | where the prices and the universe live |
|
||||||
|
| `QUANTCORE_CUOPT_URL` | `http://127.0.0.1:5000` | the GPU MILP solver (a cuOpt container on the same machine); unreachable is fine, see §9 |
|
||||||
|
| `QUANTCORE_BASE_CURRENCY` | `CHF` | reporting currency of every figure |
|
||||||
|
| `QUANTCORE_ENABLE_UI` | `true` | `false` serves the JSON API and `/docs` only |
|
||||||
|
|
||||||
|
## 4. Check that it works
|
||||||
|
|
||||||
|
```bash
|
||||||
|
quantcore-api selftest
|
||||||
|
```
|
||||||
|
|
||||||
|
This calls **every endpoint** with a small request and prints what each one answered:
|
||||||
|
|
||||||
|
```
|
||||||
|
endpoint result ms evidence
|
||||||
|
0 GET /health ok 260 745 stocks, last 2026-09-21, cuOpt up
|
||||||
|
5 GET /prices ok 28 5667 dates x 3 symbols
|
||||||
|
9 POST /risk/assets ok 10 3 rows, cvar95_hist 0.1259
|
||||||
|
21 POST /optimize (gaussian_exact) ok 63 Optimal on highs, 7 positions, CVaR 44,551
|
||||||
|
...
|
||||||
|
24 ok, 0 failed, 3 skipped in 1.0 s against http://127.0.0.1:8765
|
||||||
|
```
|
||||||
|
|
||||||
|
Exit code 0 means the deployment is sound. Run this first whenever something behaves oddly — it
|
||||||
|
separates "the service is broken" from "my request is wrong" in one second. The three skipped probes
|
||||||
|
write to the store or download data; `--write` includes them. `--full` optimises on a real scenario
|
||||||
|
set instead of the fast Gaussian mode.
|
||||||
|
|
||||||
|
`quantcore-api endpoints` lists the API surface, and `<url>/docs` is the interactive reference with
|
||||||
|
every field of every request. **That is the authoritative contract** — this guide is the tour.
|
||||||
|
|
||||||
|
## 5. Conventions you have to know
|
||||||
|
|
||||||
|
These are not negotiable per request; they are how the whole package thinks.
|
||||||
|
|
||||||
|
* **Returns are natural logarithms.** `ln(P_t / P_{t-1})`. A CHF return of a foreign stock is the
|
||||||
|
local log-return plus the FX log-return — they add.
|
||||||
|
* **Money is CHF**, always, whatever the listing currency.
|
||||||
|
* **Positions are integer share numbers**, never percentages. That is the point of the MILP: you
|
||||||
|
cannot buy 0.3 of a share, and at CHF 1 m the difference matters.
|
||||||
|
* **A horizon is `"day"`, `"week"`, `"month"`, `"quarter"`, `"year"`, or an integer number of
|
||||||
|
periods** of the chosen frequency. `"week"` at daily frequency is 5 trading days.
|
||||||
|
* **Risk is CVaR** (expected shortfall) at 95 % and 99 %, expressed as an absolute amount over the
|
||||||
|
horizon. VaR comes along for reference.
|
||||||
|
* **Two knobs on every parametric risk figure**, and the defaults are not always what you want:
|
||||||
|
* `drift`: `"estimated"` subtracts the window's mean return; `"zero"` uses volatility only. Use
|
||||||
|
`"zero"` for anything longer than a week — over a quarter the estimated mean is the noisiest
|
||||||
|
input in the calculation and it can halve the reported risk.
|
||||||
|
* `space`: `"log"` reports the figure as a log-return; `"value"` as a fraction of the position's
|
||||||
|
value (`1 - e^{-x}`), which cannot exceed 100 %. Use `"value"` in anything a client reads.
|
||||||
|
|
||||||
|
## 6. The endpoints
|
||||||
|
|
||||||
|
| method | path | what it answers |
|
||||||
|
|---|---|---|
|
||||||
|
| GET | `/health` | is the service up, what data does it hold, is the GPU solver reachable |
|
||||||
|
| GET | `/universe`, `/universe/{symbol}` | the investable stocks and their metadata |
|
||||||
|
| GET | `/data/symbols` | the universe as one table plus filter facets — feeds a stock picker |
|
||||||
|
| GET | `/prices`, `/returns` | adjusted closes and log-returns, daily/weekly/monthly, CHF or local |
|
||||||
|
| GET | `/market/index`, `/market/weights` | the cap-weighted broad market and its weights |
|
||||||
|
| POST | `/risk/assets` | per-stock volatility, VaR, CVaR (normal and historical, rolling and EWM) |
|
||||||
|
| POST | `/risk/series` | those figures through time, for a chart |
|
||||||
|
| POST | `/risk/covariance`, `/risk/correlation` | the matrices as measured |
|
||||||
|
| POST | `/risk/portfolio` | VaR/CVaR of a portfolio held in shares |
|
||||||
|
| POST | `/performance/assets`, `/performance/rolling`, `/performance/portfolio` | return, Sharpe, Sortino, drawdown, Calmar, skew, kurtosis |
|
||||||
|
| POST | `/model/expected-returns` | Black–Litterman: equilibrium π, your views, the posterior |
|
||||||
|
| POST | `/model/covariance` | the covariance the models use, with its conditioning |
|
||||||
|
| POST | `/model/scenarios` | a scenario set, its moments, a portfolio's P&L distribution |
|
||||||
|
| POST | `/optimize` | the mean–CVaR MILP: a new portfolio or a rebalancing |
|
||||||
|
| POST | `/jobs/setup`, `/jobs/update`, GET `/jobs`, `/jobs/{id}` | background data work |
|
||||||
|
| GET/POST | `/snapshots` | freeze the data so a demo stays reproducible |
|
||||||
|
|
||||||
|
## 7. From Python
|
||||||
|
|
||||||
|
```python
|
||||||
|
from quantcore.client import QuantcoreClient
|
||||||
|
|
||||||
|
qc = QuantcoreClient("http://127.0.0.1:5001") # or omit: $QUANTCORE_API_URL, else this default
|
||||||
|
|
||||||
|
qc.health()["store"]["last_price_date"] # '2026-09-21'
|
||||||
|
|
||||||
|
# what can I invest in?
|
||||||
|
swiss = qc.universe(country="Switzerland")
|
||||||
|
health = [s["symbol"] for s in qc.universe(region="Europe", sector="Health Care")]
|
||||||
|
|
||||||
|
# what is one stock's risk over a week, as a client would read it?
|
||||||
|
row = qc.asset_risk(["NESN.SW"], horizon="week", drift="zero", space="value")[0]
|
||||||
|
row["cvar95_normal_ewm"] # 0.0665 -> 6.7 % of the position in the worst 5 %
|
||||||
|
|
||||||
|
# what is my client's current portfolio risking?
|
||||||
|
qc.portfolio_risk({"NESN.SW": 500, "AAPL": 100}, cash=20_000, horizon="month")
|
||||||
|
|
||||||
|
# the research agent found something positive on Apple, with middling confidence
|
||||||
|
views = [{"symbol": "AAPL", "sentiment": 0.6, "confidence": 0.4}]
|
||||||
|
post = qc.expected_returns(max_assets=100, views=views)
|
||||||
|
|
||||||
|
# propose a portfolio for CHF 1 m that risks at most CHF 30 000 in the worst 5 % of weeks
|
||||||
|
res = qc.optimize(cash=1_000_000, horizon="week", confidence=0.95, cvar_limit=30_000,
|
||||||
|
views=views, risk_drift="zero", time_limit=300)
|
||||||
|
|
||||||
|
res["status"] # 'Optimal'
|
||||||
|
res["cvar"] # 30000.00 -- the limit binds exactly
|
||||||
|
for p in res["positions"]:
|
||||||
|
print(p["symbol"], p["shares"], round(p["weight"], 4))
|
||||||
|
```
|
||||||
|
|
||||||
|
Every method passes its keyword arguments through to the API unchanged, so the field names in
|
||||||
|
`/docs` are the field names here. A non-2xx answer raises `QuantcoreError` with the API's own
|
||||||
|
message; an unreachable server raises it too, with `unreachable = True`.
|
||||||
|
|
||||||
|
## 8. From the command line
|
||||||
|
|
||||||
|
The same surface, for looking at things and for scripting:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
quantcore-api universe --region Europe --sector "Health Care"
|
||||||
|
quantcore-api prices NESN.SW AAPL --freq weekly --tail 5
|
||||||
|
quantcore-api risk NESN.SW AAPL --horizon quarter --drift zero --space value
|
||||||
|
quantcore-api correlation NESN.SW NOVN.SW ROG.SW
|
||||||
|
quantcore-api risk-portfolio --holdings '{"NESN.SW": 500, "AAPL": 100}' --cash 20000
|
||||||
|
quantcore-api expected-returns --max-assets 50 --view AAPL:0.6:0.4
|
||||||
|
quantcore-api scenarios NESN.SW AAPL --model gbm --n-scenarios 4096
|
||||||
|
quantcore-api optimize --cash 1000000 --cvar-limit 30000 --horizon week --risk-drift zero
|
||||||
|
quantcore-api optimize --request my_request.json --json > result.json
|
||||||
|
quantcore-api update --wait # refresh prices and follow the job
|
||||||
|
```
|
||||||
|
|
||||||
|
`--view SYMBOL:SENTIMENT[:CONFIDENCE[:STRENGTH]]` is the shorthand for a sentiment view, repeatable.
|
||||||
|
`--json` prints the raw answer (pipe it into `jq`, or save it as a fixture); it works before or after
|
||||||
|
the subcommand. `--request FILE` takes a full JSON request, which is the right way to keep a
|
||||||
|
complicated setup reproducible — its fields win over the flags.
|
||||||
|
|
||||||
|
## 9. The five things the project actually needs
|
||||||
|
|
||||||
|
**Pick a universe.** `/data/symbols` gives you rows *and* the counts per region, sector and country
|
||||||
|
in one call, so a picker needs no second request. Pass the resulting symbols as `symbols` to
|
||||||
|
`/optimize` to restrict the candidates; leave it out and the largest `max_assets` by market cap are
|
||||||
|
used.
|
||||||
|
|
||||||
|
**Measure what the client holds.** `/risk/portfolio` with their share numbers and cash. Use
|
||||||
|
`method="historical"` alongside the default `"normal"` and show both — where they disagree, the
|
||||||
|
distribution is not Gaussian, and that is worth saying out loud.
|
||||||
|
|
||||||
|
**Turn an assessment into a view.** A view is a statement about an *annual expected return* with a
|
||||||
|
degree of belief. Either explicit (`{"assets": {"NESN.SW": 1.0}, "q": 0.09}`: "Nestlé will return
|
||||||
|
9 %") or from sentiment (`{"symbol": "NESN.SW", "sentiment": 0.6, "confidence": 0.4}`). Sentiment in
|
||||||
|
[−1, 1] scales `strength` (default 25 % p.a. at full sentiment); `confidence` in (0, 1] is the
|
||||||
|
Idzorek confidence that sets Ω. Call `/model/expected-returns` *before* optimising to see what your
|
||||||
|
view did to the posterior — that is the number you will have to defend.
|
||||||
|
|
||||||
|
**Propose a new portfolio.** `/optimize` with `mode="new"` and `cash`. Set `cvar_limit` (absolute
|
||||||
|
CHF) or `cvar_limit_fraction` (of wealth). Constraints live in `constraints`, costs in `costs`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"mode": "new", "cash": 1000000, "horizon": "week", "confidence": 0.95, "cvar_limit": 30000,
|
||||||
|
"risk_drift": "zero",
|
||||||
|
"constraints": {"max_weight": 0.10, "max_sector_weight": 0.25, "max_positions": 25,
|
||||||
|
"min_position_value": 5000, "min_invested": 0.95},
|
||||||
|
"costs": {"proportional_buy": 0.001, "proportional_sell": 0.001, "fixed_per_trade": 0}}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rebalance an existing one.** `mode="rebalance"` with `holdings` (shares) and the `cash` available.
|
||||||
|
Transaction costs are charged and the result carries a `before` block, so you can show the client
|
||||||
|
both states and what the trade bought them:
|
||||||
|
|
||||||
|
```python
|
||||||
|
res = qc.optimize(mode="rebalance", holdings={"NESN.SW": 500, "AAPL": 100}, cash=50_000,
|
||||||
|
horizon="week", cvar_limit=30_000, risk_drift="zero",
|
||||||
|
constraints={"max_turnover": 200_000})
|
||||||
|
res["before"]["cvar"], res["cvar"] # risk before and after
|
||||||
|
res["total_cost"], res["turnover"] # what the change costs
|
||||||
|
[p for p in res["positions"] if p["trade"] != 0] # the trades to place
|
||||||
|
```
|
||||||
|
|
||||||
|
## 10. Reading a result
|
||||||
|
|
||||||
|
Every optimisation answers with its own provenance. The fields you will quote:
|
||||||
|
|
||||||
|
| field | meaning |
|
||||||
|
|---|---|
|
||||||
|
| `status` | `Optimal` \| `FeasibleFound` (time limit — check the gap) \| `Infeasible` \| `Error` |
|
||||||
|
| `solver` | `cuopt` (GPU) or `highs` (CPU) — which one actually produced this |
|
||||||
|
| `reproducibility` | `exact` (HiGHS) \| `within_gap` (cuOpt, 1e-4) \| `none` |
|
||||||
|
| `expected_pnl`, `expected_return` | over the horizon, in CHF and as a fraction |
|
||||||
|
| `var`, `cvar`, `cvar_limit`, `confidence` | the risk figures and the limit they had to respect |
|
||||||
|
| `positions` | per stock: `shares`, `shares_before`, `trade`, `value`, `weight`, `cost`, `expected_return_ann` |
|
||||||
|
| `sector_exposure`, `invested`, `cash_end`, `total_cost`, `turnover` | the resulting portfolio |
|
||||||
|
| `model` | covariance estimator, scenario model and seed, expected-return model, the views and the Ω that was calibrated from them |
|
||||||
|
| `scenario_key`, `scenario_hash` | what the scenarios depend on, and a fingerprint of the draws |
|
||||||
|
| `warnings` | **read these** — see below |
|
||||||
|
| `timings` | seconds per stage |
|
||||||
|
|
||||||
|
So a sentence in your advisory output can be backed end to end: *"CVaR 95 % = CHF 29 999 over one
|
||||||
|
week (mean–CVaR MILP, EWM λ = 0.99, GBM scenarios, seed 42, solved to optimality on HiGHS)."*
|
||||||
|
|
||||||
|
**`warnings` is not decoration.** It is where the service tells you that the answer is technically
|
||||||
|
correct and economically doubtful: a risk limit that could not bind, a covariance estimate too
|
||||||
|
singular for the number of candidates, too few scenarios per candidate, an expected return above
|
||||||
|
60 % p.a. on a stock it decided to hold. Surface them or handle them; do not drop them.
|
||||||
|
|
||||||
|
**Determinism.** The same request gives the same scenarios (`scenario_hash`) and the same portfolio,
|
||||||
|
on the same machine. Across machines — a different numpy or pandas — the `scenario_key` still
|
||||||
|
matches while the digest may not, because floating-point results agree to 13–14 digits rather than
|
||||||
|
bit for bit; the figures agree to well under a franc. Compare runs by `scenario_key`. For a demo
|
||||||
|
that must be identical next week, freeze the data with a snapshot (`POST /snapshots`).
|
||||||
|
|
||||||
|
## 11. Things that will bite you
|
||||||
|
|
||||||
|
* **The service answers one request at a time while a solve runs.** The MILP holds the interpreter.
|
||||||
|
Do not build a UI that fires an optimisation and then polls something else — it will look hung.
|
||||||
|
* **`time_limit` is the budget of the whole request**, not of each solver call (row generation and
|
||||||
|
the Gaussian mode solve a *sequence* of MILPs). Without it, a large candidate set on HiGHS can run
|
||||||
|
for hours. Set it. 300 s is a sane default.
|
||||||
|
* **The GPU solver is optional.** `solver="auto"` uses cuOpt if it answers and HiGHS otherwise; the
|
||||||
|
result says which, and adds a warning. A cuOpt outage degrades speed, never correctness.
|
||||||
|
* **`risk_drift="model"` (the default) measures the CVaR on scenarios that include the expected
|
||||||
|
return.** At long horizons the drift grows with *h* while the spread grows with *√h*, so a bullish
|
||||||
|
forecast can push the whole loss tail above zero and the risk limit stops binding — the optimiser
|
||||||
|
will then happily propose something absurd. Use `risk_drift="zero"` for anything beyond a few
|
||||||
|
weeks; the objective still maximises expected return, but the risk measurement no longer believes
|
||||||
|
the forecast.
|
||||||
|
* **Scenario count drives stability.** The default is 16 per candidate (within 4 096…16 384). The
|
||||||
|
reported figures are stable from ~2 000 on, but the *composition* moves between near-equivalent
|
||||||
|
portfolios; about 64 per candidate pins it. The result warns below 8.
|
||||||
|
* **`scenario_model="gaussian_exact"` is fast but bounds the CVaR from below.** It replaces the
|
||||||
|
scenarios with linear cuts, and until those converge the limit can be overshot by a hair (a franc
|
||||||
|
or two on a million, reported in `warnings`). Use it to explore; use `"gbm"` or `"empirical"` when
|
||||||
|
the limit has to hold exactly.
|
||||||
|
* **`Infeasible` means your constraints contradict each other**, most often `min_invested` against a
|
||||||
|
tight `cvar_limit`, or a `max_turnover` that cannot reach the risk budget. Relax one and re-run.
|
||||||
|
|
||||||
|
## 12. Troubleshooting
|
||||||
|
|
||||||
|
| symptom | what it means |
|
||||||
|
|---|---|
|
||||||
|
| `cannot reach the quant-core API at ...` | no service there. `quantcore serve`, or fix `--url` / `QUANTCORE_API_URL` |
|
||||||
|
| `503 no market data yet` | the store is empty: `POST /jobs/setup` (~15 min) or mount a prepared data directory |
|
||||||
|
| `422` with a field name | your request does not match the schema; the message names the field. `/docs` has the valid values |
|
||||||
|
| `/health` shows `cuopt.reachable: false` | the GPU solver is down; optimisations fall back to HiGHS and say so |
|
||||||
|
| `FeasibleFound` instead of `Optimal` | the time limit hit; raise `time_limit`, or lower `n_scenarios` / `max_assets` |
|
||||||
|
| an optimisation "seems stuck" | it is solving, and the service is single-threaded. `timings` afterwards shows where the time went |
|
||||||
|
| numbers differ slightly from a colleague's | different environment, same inputs: compare `scenario_key`, not `scenario_hash` (§10) |
|
||||||
|
| Yahoo errors during a data job | rate limiting; the client retries and opens a circuit. Re-run later, the stored data is intact |
|
||||||
|
|
||||||
|
Everything failing at once? `quantcore-api selftest` tells you whether it is the service or you.
|
||||||
|
|
||||||
|
## 13. Where the details are
|
||||||
|
|
||||||
|
| document | what is in it |
|
||||||
|
|---|---|
|
||||||
|
| `<url>/docs` | every endpoint and field, interactive — the contract |
|
||||||
|
| `README.md` | the services, the library API, quick start |
|
||||||
|
| `docs/ARCHITECTURE.md` | how the pieces fit, what depends on what, where the AI layer plugs in |
|
||||||
|
| `docs/METHODOLOGY.md` | every formula, estimator and convention, with the measured numbers |
|
||||||
|
| `docs/optimization_model.pdf` | the MILP derived from Rockafellar–Uryasev to whole shares, and the sampling theory |
|
||||||
|
| `docs/measurement_and_models.pdf` | risk and performance measures, GBM and empirical innovations, Black–Litterman with Idzorek confidence |
|
||||||
BIN
docs/measurement_and_models.pdf
Normal file
BIN
docs/measurement_and_models.pdf
Normal file
Binary file not shown.
807
docs/measurement_and_models.tex
Normal file
807
docs/measurement_and_models.tex
Normal file
@ -0,0 +1,807 @@
|
|||||||
|
% !TEX encoding = UTF-8 Unicode
|
||||||
|
% ---------------------------------------------------------------------------------------------
|
||||||
|
% quantcore -- documentation of the measurement layer, the scenario models and the
|
||||||
|
% expected-return model, as implemented in quantcore/measurement and quantcore/models.
|
||||||
|
% Build: latexmk -pdf docs/measurement_and_models.tex
|
||||||
|
% ---------------------------------------------------------------------------------------------
|
||||||
|
\documentclass[11pt,a4paper]{article}
|
||||||
|
\usepackage[utf8]{inputenc}
|
||||||
|
\usepackage[T1]{fontenc}
|
||||||
|
\usepackage[british]{babel}
|
||||||
|
\usepackage{lmodern}
|
||||||
|
\usepackage{microtype}
|
||||||
|
\usepackage[margin=2.4cm]{geometry}
|
||||||
|
\usepackage{amsmath,amssymb,amsthm,mathtools}
|
||||||
|
\usepackage{booktabs,array}
|
||||||
|
\usepackage{enumitem}
|
||||||
|
\usepackage{xcolor}
|
||||||
|
\usepackage[colorlinks=true,linkcolor=blue!50!black,citecolor=blue!50!black,urlcolor=blue!50!black]{hyperref}
|
||||||
|
|
||||||
|
\newtheorem{theorem}{Theorem}[section]
|
||||||
|
\newtheorem{proposition}[theorem]{Proposition}
|
||||||
|
\theoremstyle{definition}
|
||||||
|
\newtheorem{definition}[theorem]{Definition}
|
||||||
|
\newtheorem{example}[theorem]{Example}
|
||||||
|
\theoremstyle{remark}
|
||||||
|
\newtheorem{remark}[theorem]{Remark}
|
||||||
|
|
||||||
|
\DeclareMathOperator{\CVaR}{CVaR}
|
||||||
|
\DeclareMathOperator{\VaR}{VaR}
|
||||||
|
\DeclareMathOperator{\E}{\mathbb{E}}
|
||||||
|
\DeclareMathOperator{\Var}{Var}
|
||||||
|
\DeclareMathOperator{\Cov}{Cov}
|
||||||
|
\DeclareMathOperator{\diag}{diag}
|
||||||
|
\DeclareMathOperator{\tr}{tr}
|
||||||
|
\newcommand{\R}{\mathbb{R}}
|
||||||
|
\newcommand{\pos}[1]{\left[#1\right]^{+}}
|
||||||
|
\newcommand{\code}[1]{\texttt{\hyphenchar\font=-1 #1}}
|
||||||
|
\newcommand{\bs}{\boldsymbol}
|
||||||
|
\setlength{\parskip}{4pt}
|
||||||
|
\setlength{\parindent}{0pt}
|
||||||
|
|
||||||
|
\title{Risk and Performance Measurement, Scenario Generation\\ and the Expected-Return Model\\[1ex]
|
||||||
|
\large From log-returns to VaR/CVaR, from the covariance estimators to simulated horizon
|
||||||
|
scenarios, and from market capitalisations to Black--Litterman expected returns with
|
||||||
|
Idzorek confidence views, as implemented in \code{quantcore}}
|
||||||
|
\author{AISE502 -- AI-Augmented Portfolio Intelligence Platform\\ \small Deterministic quant core, modules \code{quantcore.measurement} and \code{quantcore.models}}
|
||||||
|
\date{September 2026}
|
||||||
|
|
||||||
|
\begin{document}
|
||||||
|
\maketitle
|
||||||
|
|
||||||
|
\begin{abstract}
|
||||||
|
\noindent
|
||||||
|
This note is the companion of \emph{Mean--CVaR Portfolio Optimisation in Whole Shares}
|
||||||
|
(the file \code{optimization\_model.pdf} in \code{docs/}). It documents the three model layers that feed the
|
||||||
|
optimiser and the platform's risk and performance figures: (i) the \emph{measurement} layer ---
|
||||||
|
log-returns in CHF, moving-window and exponentially weighted estimators of mean, volatility and
|
||||||
|
covariance, Value-at-Risk and Conditional Value-at-Risk at 95\,\% and 99\,\% for a day, a week
|
||||||
|
or a month, and the performance statistics; (ii) the \emph{stock-price models} that turn the
|
||||||
|
latest covariance into a set of horizon scenarios --- geometric Brownian motion and the
|
||||||
|
empirical-innovation model $\varepsilon_t=\Sigma_{t-1}^{-1/2}r_t$, $r(j\,|\,T)=\Sigma_T^{1/2}\varepsilon_j$;
|
||||||
|
and (iii) the \emph{expected-return model} --- market-capitalisation weights, the broad-market
|
||||||
|
average, reverse optimisation, and the Black--Litterman posterior with views whose uncertainty is
|
||||||
|
calibrated from a degree of belief following Idzorek. Every formula is stated with its
|
||||||
|
conventions and defaults, the numerical safeguards (covariance shrinkage, innovation
|
||||||
|
re-standardisation) are motivated with measurements on the platform's own data set, and the last
|
||||||
|
section maps the mathematics to the functions, parameters and output fields of the code.
|
||||||
|
\end{abstract}
|
||||||
|
|
||||||
|
\tableofcontents
|
||||||
|
|
||||||
|
% ============================================================================================
|
||||||
|
\section{Data, conventions and notation}
|
||||||
|
% ============================================================================================
|
||||||
|
|
||||||
|
The universe consists of $N$ stocks with adjusted closing prices $P_i(t)$ in their listing
|
||||||
|
currency $c(i)$, observed on the \emph{union calendar} of all exchanges (a day on which at least
|
||||||
|
one exchange in the universe trades). A holiday of one exchange is filled with that exchange's
|
||||||
|
previous close (at most five days), so every trading day of the union calendar has a price for
|
||||||
|
every stock. Prices that are cleaned away (broken adjustments, spike ticks; see
|
||||||
|
\code{docs/METHODOLOGY.md}, Section~1) are treated the same way.
|
||||||
|
|
||||||
|
FX rates $X_c(t)$ are quoted as CHF per one unit of currency $c$ (so $X_{\mathrm{CHF}}\equiv1$),
|
||||||
|
and the base currency of everything below is CHF.
|
||||||
|
|
||||||
|
Three frequencies are used: \emph{daily} (union calendar, 252 periods per year), \emph{weekly}
|
||||||
|
(Friday closes, 52) and \emph{monthly} (month-end closes, 12); $\pi$ denotes the number of
|
||||||
|
periods per year of the chosen frequency. A \emph{horizon} is expressed as a number $h$ of
|
||||||
|
periods: at daily frequency a day is $h=1$, a week $h=5$, a month $h=21$; at weekly frequency
|
||||||
|
a week is $h=1$ and a month $h=4$.
|
||||||
|
|
||||||
|
Throughout, $r$ denotes a log-return, $R=e^{r}-1$ the corresponding simple return, and
|
||||||
|
\emph{losses are positive numbers}. Vectors are columns; $\mathbf 1$ is the vector of ones.
|
||||||
|
|
||||||
|
% ============================================================================================
|
||||||
|
\section{Returns}
|
||||||
|
% ============================================================================================
|
||||||
|
|
||||||
|
\subsection{Log-returns and the currency conversion}
|
||||||
|
|
||||||
|
The one-period log-return of stock $i$ in its listing currency is
|
||||||
|
\begin{equation}
|
||||||
|
r_i(t) \;=\; \ln P_i(t) - \ln P_i(t-1).
|
||||||
|
\label{eq:logret}
|
||||||
|
\end{equation}
|
||||||
|
The price in CHF is $P_i^{\mathrm{CHF}}(t)=P_i(t)\,X_{c(i)}(t)$, hence its log-return is the
|
||||||
|
\emph{sum} of the local return and the FX return,
|
||||||
|
\begin{equation}
|
||||||
|
r_i^{\mathrm{CHF}}(t) \;=\; r_i(t) + \big(\ln X_{c(i)}(t) - \ln X_{c(i)}(t-1)\big),
|
||||||
|
\label{eq:chfret}
|
||||||
|
\end{equation}
|
||||||
|
exactly and without any cross term --- the reason log-returns are the working unit of the whole
|
||||||
|
platform. (With simple returns the conversion is $(1+R_i)(1+R_X)-1$, which is neither additive
|
||||||
|
nor symmetric.) The implementation converts prices with the FX frame and takes the log-return of
|
||||||
|
the product; \eqref{eq:chfret} is checked by a unit test.
|
||||||
|
|
||||||
|
\subsection{Aggregation over time and over assets}
|
||||||
|
|
||||||
|
Under the i.i.d.\ assumption of the price models (Section~\ref{sec:models}) log-returns add over
|
||||||
|
time: the $h$-period return is $r^{(h)}(t)=\sum_{k=0}^{h-1} r(t-k)$. The historical risk
|
||||||
|
measures use \emph{overlapping} $h$-period sums (more observations, mildly autocorrelated);
|
||||||
|
non-overlapping blocks are available as an option.
|
||||||
|
|
||||||
|
Over assets, log-returns do not add. The exact log-return of a portfolio whose value weights
|
||||||
|
$w_i$ are fixed at the start of the period is
|
||||||
|
\begin{equation}
|
||||||
|
r_p \;=\; \ln\Big(1+\sum_i w_i\,(e^{r_i}-1)\Big),
|
||||||
|
\label{eq:portret}
|
||||||
|
\end{equation}
|
||||||
|
because simple returns aggregate linearly. A cash residual $1-\mathbf 1^\top w$ earns zero. This
|
||||||
|
formula (\code{portfolio\_log\_returns}) is used wherever a portfolio return series is needed
|
||||||
|
--- the historical portfolio VaR and the portfolio performance statistics.
|
||||||
|
|
||||||
|
\subsection{Why log-returns are annualised additively}
|
||||||
|
|
||||||
|
Means of log-returns scale with time: $\E[r^{(h)}]=h\mu$, and under independence
|
||||||
|
$\Var[r^{(h)}]=h\sigma^2$. An annualised log-return is therefore $\pi\mu$, an annualised
|
||||||
|
volatility $\sqrt{\pi}\,\sigma$; the geometric (compounded) annual return is $e^{\pi\mu}-1$. Both
|
||||||
|
are reported.
|
||||||
|
|
||||||
|
% ============================================================================================
|
||||||
|
\section{Moving-window and exponentially weighted estimators}
|
||||||
|
\label{sec:estimators}
|
||||||
|
% ============================================================================================
|
||||||
|
|
||||||
|
Every statistic exists in two flavours (\code{quantcore.measurement.estimators}).
|
||||||
|
|
||||||
|
\subsection{Moving window}
|
||||||
|
|
||||||
|
With window length $w$ (default 200 periods) the estimates at date $t$ use the observations
|
||||||
|
$t-w+1,\dots,t$ with equal weight $1/w$: the sample mean $m_t$, the sample standard deviation
|
||||||
|
(divisor $w-1$), the sample covariance
|
||||||
|
\begin{equation}
|
||||||
|
\Sigma_t^{\mathrm{roll}} \;=\; \frac{1}{w-1}\sum_{k=0}^{w-1}(r_{t-k}-m_t)(r_{t-k}-m_t)^\top .
|
||||||
|
\label{eq:rollcov}
|
||||||
|
\end{equation}
|
||||||
|
Rows with a missing value are dropped for matrices; a series needs at least $w/2$ observations
|
||||||
|
before a value is reported.
|
||||||
|
|
||||||
|
\subsection{Exponential weighting}
|
||||||
|
|
||||||
|
With decay $\lambda\in(0,1)$ the observation $k$ periods back receives the weight
|
||||||
|
$\lambda^k$ (RiskMetrics \cite{RiskMetrics1996}); the weights are normalised over the available
|
||||||
|
history,
|
||||||
|
\begin{equation}
|
||||||
|
\omega_k \;=\; \frac{\lambda^k}{\sum_{j=0}^{n-1}\lambda^j}, \qquad k=0,\dots,n-1,
|
||||||
|
\label{eq:ewmw}
|
||||||
|
\end{equation}
|
||||||
|
(pandas \code{ewm(alpha=1-$\lambda$, adjust=True)}), and the estimators are the weighted
|
||||||
|
moments
|
||||||
|
\begin{equation}
|
||||||
|
m_t^{\mathrm{ewm}} = \sum_k\omega_k\, r_{t-k}, \qquad
|
||||||
|
\Sigma_t^{\mathrm{ewm}} = \sum_k \omega_k\,(r_{t-k}-m_t^{\mathrm{ewm}})(r_{t-k}-m_t^{\mathrm{ewm}})^\top,
|
||||||
|
\label{eq:ewmcov}
|
||||||
|
\end{equation}
|
||||||
|
with the diagonal of \eqref{eq:ewmcov} as the EWM variance (\code{var(bias=True)}). Both
|
||||||
|
estimators are \emph{centred}: the corresponding mean is subtracted. The effective window is
|
||||||
|
$1/(1-\lambda)$ observations (half-life $\ln 0.5/\ln\lambda$):
|
||||||
|
|
||||||
|
\begin{center}
|
||||||
|
\small
|
||||||
|
\begin{tabular}{@{}lcccc@{}}
|
||||||
|
\toprule
|
||||||
|
$\lambda$ & 0.94 & 0.97 & 0.99 & 0.997 \\
|
||||||
|
\midrule
|
||||||
|
effective window $1/(1-\lambda)$ & 17 & 33 & 100 & 333 \\
|
||||||
|
half-life (periods) & 11.2 & 22.8 & 69.0 & 230.7 \\
|
||||||
|
weight of the observation 100 periods back & 0.002 & 0.048 & 0.366 & 0.741 \\
|
||||||
|
\bottomrule
|
||||||
|
\end{tabular}
|
||||||
|
\end{center}
|
||||||
|
|
||||||
|
$\lambda=0.94$ is the RiskMetrics value for daily data and the default of the measurement
|
||||||
|
services (fast adaptation, noisy); $\lambda=0.99$ is the default of the optimisation covariance
|
||||||
|
(a smoother matrix that still adapts within a quarter). For matrices the history is truncated
|
||||||
|
where the weights fall below $10^{-4}$ (917 observations for $\lambda=0.99$).
|
||||||
|
|
||||||
|
\subsection{Sequences of matrices}
|
||||||
|
|
||||||
|
The innovation filter of Section~\ref{sec:fhs} needs $\Sigma_{t-1}$ for \emph{every} date $t$,
|
||||||
|
i.e.\ the covariance estimated from the data up to and including $t-1$. Storing $T$ matrices of
|
||||||
|
size $N\times N$ is impossible for a thousand stocks, so \code{covariance\_sequence} streams
|
||||||
|
them: the rolling estimator recomputes \eqref{eq:rollcov} for each date ($O(wN^2)$ per step),
|
||||||
|
while the exponentially weighted estimator uses the exact recursion of the centred filter with
|
||||||
|
$\alpha=1-\lambda$,
|
||||||
|
\begin{equation}
|
||||||
|
d_t = r_t - m_{t-1}, \qquad m_t = m_{t-1} + \alpha\, d_t, \qquad
|
||||||
|
\Sigma_t = \lambda\,\Sigma_{t-1} + \alpha\, d_t\,(r_t-m_t)^\top ,
|
||||||
|
\label{eq:ewmrec}
|
||||||
|
\end{equation}
|
||||||
|
which costs $O(N^2)$ per step and, after a warm-up of $\max(50, 2/(1-\lambda))$ observations
|
||||||
|
initialised with \eqref{eq:ewmcov}, coincides with the normalised-weight estimate up to the
|
||||||
|
factor $1/(1-\lambda^n)\to1$ (verified by a unit test at 5\,\% for $\lambda=0.97$).
|
||||||
|
|
||||||
|
\subsection{Shrinkage towards the diagonal}
|
||||||
|
\label{sec:shrinkage}
|
||||||
|
|
||||||
|
A covariance estimated from $n$ effective observations is ill-conditioned when $N$ is not small
|
||||||
|
against $n$ and singular for $N>n$: the smallest eigenvalues are underestimated, the largest
|
||||||
|
overestimated (the Marchenko--Pastur effect), and anything that inverts the matrix --- the
|
||||||
|
innovation filter, mean-variance weights --- amplifies the error. The classical remedy is
|
||||||
|
linear shrinkage \cite{LedoitWolf2004} towards a structured target; the platform shrinks towards
|
||||||
|
the diagonal (the target of Schäfer and Strimmer \cite{SchaeferStrimmer2005}), which keeps the
|
||||||
|
variances and pulls the correlations towards zero:
|
||||||
|
\begin{equation}
|
||||||
|
\Sigma^{(s)} \;=\; (1-s)\,\Sigma + s\,\diag(\Sigma), \qquad s\in[0,1].
|
||||||
|
\label{eq:shrink}
|
||||||
|
\end{equation}
|
||||||
|
The intensity can be given, or estimated from the data (\code{shrinkage="auto"}): for centred
|
||||||
|
observations $x_k$ ($k=1,\dots,n$), products $w_{kij}=x_{ki}x_{kj}$ and sample covariances
|
||||||
|
$s_{ij}=\frac{n}{n-1}\bar w_{ij}$, the intensity that minimises the expected squared Frobenius
|
||||||
|
loss of \eqref{eq:shrink} is
|
||||||
|
\begin{equation}
|
||||||
|
s^\ast \;=\; \frac{\sum_{i\ne j}\widehat{\Var}(s_{ij})}{\sum_{i\ne j}s_{ij}^2},
|
||||||
|
\qquad
|
||||||
|
\widehat{\Var}(s_{ij}) \;=\; \frac{n}{(n-1)^3}\sum_{k=1}^n \big(w_{kij}-\bar w_{ij}\big)^2 ,
|
||||||
|
\label{eq:ss}
|
||||||
|
\end{equation}
|
||||||
|
clipped to $[0,1]$. It is small for a long, strongly correlated sample and approaches one when
|
||||||
|
$N$ approaches $n$ --- exactly when the sample covariance becomes unreliable. For the EWM
|
||||||
|
estimator the intensity is computed on the last $3/(1-\lambda)$ observations with equal weights.
|
||||||
|
On the platform's data set (daily CHF returns, $\lambda=0.99$) the automatic intensity is 0.047
|
||||||
|
for the 100 largest stocks, 0.069 for 300 and 0.114 for the full universe of 745 names. The
|
||||||
|
sequence \eqref{eq:ewmrec} uses one intensity for all its matrices.
|
||||||
|
|
||||||
|
% ============================================================================================
|
||||||
|
\section{Risk measures}
|
||||||
|
\label{sec:risk}
|
||||||
|
% ============================================================================================
|
||||||
|
|
||||||
|
\subsection{Loss, VaR and CVaR}
|
||||||
|
|
||||||
|
For a position of value $V_0$ and a horizon log-return $r$, the loss in return space is
|
||||||
|
$\ell=-r$ and in money it is $V_0(1-e^{r})=V_0(1-e^{-\ell})$. The Value-at-Risk at level
|
||||||
|
$\beta$ is the $\beta$-quantile of the loss distribution, the Conditional Value-at-Risk (expected
|
||||||
|
shortfall) the mean of the loss beyond it,
|
||||||
|
\begin{equation}
|
||||||
|
\VaR_\beta = q_\beta(\ell), \qquad
|
||||||
|
\CVaR_\beta = \frac{1}{1-\beta}\int_\beta^1 q_\gamma(\ell)\,d\gamma
|
||||||
|
\;\;\big(=\E[\ell\mid\ell\ge\VaR_\beta]\text{ for continuous }\ell\big),
|
||||||
|
\label{eq:varcvar}
|
||||||
|
\end{equation}
|
||||||
|
see \cite{AcerbiTasche2002,MFE2015}. The platform reports both at $\beta=0.95$ and $0.99$.
|
||||||
|
Because $\ell\mapsto V_0(1-e^{-\ell})$ is increasing, quantiles commute with it, and a
|
||||||
|
return-space VaR translates into money exactly as $V_0(1-e^{-\VaR_\beta})$; the CVaR in money
|
||||||
|
is computed the same way from the return-space CVaR (which is exact for VaR and a very good
|
||||||
|
approximation for CVaR: for a weekly CVaR of 4.6\,\% the difference between the exact tail mean
|
||||||
|
of $V_0(1-e^{-\ell})$ and $V_0(1-e^{-\CVaR})$ is below 0.1\,\% of $V_0$; the scenario-based
|
||||||
|
portfolio risk of Section~\ref{sec:portfoliorisk} works in money directly and is exact).
|
||||||
|
|
||||||
|
\subsection{Gaussian (GBM) VaR and CVaR}
|
||||||
|
|
||||||
|
If $r\sim N(\mu,\sigma^2)$ then $\ell=-r\sim N(-\mu,\sigma^2)$, its $\beta$-quantile is
|
||||||
|
$-\mu+z_\beta\sigma$ with $z_\beta=\Phi^{-1}(\beta)$, and the tail mean follows from
|
||||||
|
$\E[Z\mid Z\ge z_\beta]=\varphi(z_\beta)/(1-\beta)$ for a standard normal $Z$:
|
||||||
|
\begin{equation}
|
||||||
|
\VaR_\beta = -\mu + z_\beta\,\sigma, \qquad
|
||||||
|
\CVaR_\beta = -\mu + \sigma\,\frac{\varphi(z_\beta)}{1-\beta}.
|
||||||
|
\label{eq:normal}
|
||||||
|
\end{equation}
|
||||||
|
The multipliers are $z_{0.95}=1.645$, $\varphi(z_{0.95})/0.05=2.063$, $z_{0.99}=2.326$,
|
||||||
|
$\varphi(z_{0.99})/0.01=2.665$. Under the i.i.d.\ assumption the horizon parameters are
|
||||||
|
$\mu_h=h\mu$ and $\sigma_h=\sqrt h\,\sigma$; $\mu$ and $\sigma$ come from either estimator of
|
||||||
|
Section~\ref{sec:estimators}. The mean is kept (a positive drift lowers the loss quantile
|
||||||
|
one-for-one); for daily data it is negligible, for a month it is not.
|
||||||
|
|
||||||
|
\begin{example}[Reference values]
|
||||||
|
A stock with daily volatility $\sigma=1\,\%$ and zero mean, one week ($h=5$,
|
||||||
|
$\sigma_h=2.236\,\%$): $\VaR_{0.95}=3.678\,\%$, $\CVaR_{0.95}=4.612\,\%$, $\VaR_{0.99}=5.202\,\%$,
|
||||||
|
$\CVaR_{0.99}=5.960\,\%$. For a position of CHF~100\,000 the exact money figures are
|
||||||
|
$\VaR_{0.95}=3\,611$, $\CVaR_{0.95}=4\,508$, $\VaR_{0.99}=5\,069$, $\CVaR_{0.99}=5\,786$ (the
|
||||||
|
linear approximation $V_0\cdot\CVaR$ would give 4\,612 and 5\,960). The daily figures
|
||||||
|
$\VaR_{0.95}=1.645\,\%$, $\CVaR_{0.95}=2.063\,\%$, $\VaR_{0.99}=2.326\,\%$, $\CVaR_{0.99}=2.665\,\%$
|
||||||
|
and the weekly ones are pinned by \code{tests/test\_risk.py}.
|
||||||
|
\end{example}
|
||||||
|
|
||||||
|
\subsection{Historical VaR and CVaR}
|
||||||
|
|
||||||
|
The empirical counterparts replace the Gaussian by the empirical distribution of the realised
|
||||||
|
$h$-period returns $r^{(h)}$ over a window of $\max(5w,750)$ periods:
|
||||||
|
\begin{equation}
|
||||||
|
\VaR_\beta^{\mathrm{hist}} = -\hat q_{1-\beta}\big(r^{(h)}\big), \qquad
|
||||||
|
\CVaR_\beta^{\mathrm{hist}} = -\frac{1}{|\mathcal T|}\sum_{t\in\mathcal T} r^{(h)}_t, \quad
|
||||||
|
\mathcal T=\big\{t: r^{(h)}_t\le \hat q_{1-\beta}\big\},
|
||||||
|
\label{eq:hist}
|
||||||
|
\end{equation}
|
||||||
|
with $\hat q$ the empirical quantile (linear interpolation). They capture fat tails and skewness
|
||||||
|
that the Gaussian misses, at the price of noise: at $\beta=0.99$ with 750 daily observations the
|
||||||
|
tail contains eight points.
|
||||||
|
|
||||||
|
\subsection{Discrete scenario sets}
|
||||||
|
|
||||||
|
For $S$ equiprobable losses (a simulated scenario set, Section~\ref{sec:models}) the definition
|
||||||
|
\eqref{eq:varcvar} evaluates exactly: with $k=(1-\beta)S$ and the losses sorted decreasingly
|
||||||
|
$\ell_{(1)}\ge\ell_{(2)}\ge\dots$,
|
||||||
|
\begin{equation}
|
||||||
|
\CVaR_\beta = \frac1k\Big(\sum_{s=1}^{\lfloor k\rfloor}\ell_{(s)} + (k-\lfloor k\rfloor)\,\ell_{(\lfloor k\rfloor+1)}\Big),
|
||||||
|
\qquad \VaR_\beta = \ell_{(\lceil k\rceil)} ,
|
||||||
|
\label{eq:scen}
|
||||||
|
\end{equation}
|
||||||
|
which is the value the Rockafellar--Uryasev linear programme of the optimisation returns
|
||||||
|
(function \code{scenario\_cvar}; see Remark~2.3 of the companion note). For $S=2\,000$ and
|
||||||
|
$\beta=0.99$ the tail has $k=20$ scenarios.
|
||||||
|
|
||||||
|
\subsection{Per-asset tables and time series}
|
||||||
|
|
||||||
|
\code{RiskService.asset\_risk} returns, for a list of stocks, a frequency, a horizon, a window
|
||||||
|
$w$ and a decay $\lambda$, one row per stock with the annualised rolling and EWM volatilities and,
|
||||||
|
for each confidence level $\beta\in\{0.95,0.99\}$, the Gaussian VaR and CVaR from the rolling
|
||||||
|
estimator, the same from the EWM estimator, and the historical VaR and CVaR --- six figures per
|
||||||
|
level, all in log-return units over the horizon (Table~\ref{tab:impl}). \code{risk\_series} gives
|
||||||
|
the same quantities as time series: Gaussian figures from the rolling / EWM volatility and mean
|
||||||
|
series, historical ones from rolling quantiles and rolling tail means of the one-period returns,
|
||||||
|
scaled by $\sqrt h$ for longer horizons (an approximation used only in the series).
|
||||||
|
|
||||||
|
\subsection{Portfolio risk in money}
|
||||||
|
\label{sec:portfoliorisk}
|
||||||
|
|
||||||
|
A portfolio is given in \emph{numbers of shares} $n_i$ plus cash $C$. With today's CHF prices
|
||||||
|
$P_i$ the value is $V_0=C+\sum_iP_in_i$ and the value weights are $w_i=P_in_i/V_0$ (the cash
|
||||||
|
weight $C/V_0$ is riskless). Three methods are offered (\code{portfolio\_risk}):
|
||||||
|
\begin{itemize}[nosep]
|
||||||
|
\item \textbf{normal}: $\sigma_p^2=w^\top\Sigma w$ with the rolling or EWM covariance,
|
||||||
|
$\mu_p=w^\top m$ with the window mean, then \eqref{eq:normal} with $h\mu_p$ and
|
||||||
|
$\sqrt h\sigma_p$, converted to money;
|
||||||
|
\item \textbf{historical}: the realised portfolio return series \eqref{eq:portret} over the
|
||||||
|
last $\max(5w,750)$ periods, aggregated to $h$ periods, then \eqref{eq:hist};
|
||||||
|
\item \textbf{scenarios}: money P\&L $\Pi_s=V_0\sum_iw_i(e^{r_{s,i}}-1)$ for each scenario of a
|
||||||
|
simulated set, then \eqref{eq:scen} on the losses $-\Pi_s$ --- exact in money, and
|
||||||
|
precisely the risk the optimiser constrains.
|
||||||
|
\end{itemize}
|
||||||
|
The result (\code{PortfolioRisk}) carries the value, the weights, the horizon, the method, the
|
||||||
|
annualised portfolio volatility, and per confidence level the VaR and CVaR in return space and in
|
||||||
|
money. A unit test checks that Gaussian scenarios with the same mean and covariance reproduce the
|
||||||
|
normal figures within sampling error.
|
||||||
|
|
||||||
|
% ============================================================================================
|
||||||
|
\section{Performance measures}
|
||||||
|
% ============================================================================================
|
||||||
|
|
||||||
|
All statistics (\code{performance\_table}) are computed from one-period log-returns $r_t$,
|
||||||
|
$t=1,\dots,T$, at the chosen frequency with $\pi$ periods per year and the annual CHF risk-free
|
||||||
|
rate $r_f$ (0 by default):
|
||||||
|
\begin{align}
|
||||||
|
\text{cumulative return} &= e^{\sum_t r_t}-1, \quad
|
||||||
|
\text{ann.\ log-return} = \pi\,\bar r, \quad
|
||||||
|
\text{ann.\ return} = e^{\pi\bar r}-1, \notag\\
|
||||||
|
\text{ann.\ volatility} &= \sqrt{\pi}\,\hat\sigma, \quad
|
||||||
|
\text{Sharpe} = \frac{\pi\bar r - r_f}{\sqrt\pi\,\hat\sigma}, \quad
|
||||||
|
\text{Calmar} = \frac{e^{\pi\bar r}-1}{-\mathrm{MDD}}, \notag\\
|
||||||
|
\text{Sortino} &= \frac{\pi\bar r - r_f}{\sqrt{\pi\cdot\frac1T\sum_t \min(r_t,0)^2}},
|
||||||
|
\label{eq:perf}
|
||||||
|
\end{align}
|
||||||
|
where $\hat\sigma$ is the sample standard deviation (divisor $T-1$) and the maximum drawdown
|
||||||
|
$\mathrm{MDD}=\min_t\big(V_t/\max_{s\le t}V_s-1\big)$ is taken on the value path
|
||||||
|
$V_t=\exp(\sum_{s\le t}r_s)$. Skewness and excess kurtosis of $r_t$, the hit ratio (share of
|
||||||
|
positive periods), the best and the worst period, the number of observations and the start and
|
||||||
|
end dates complete the row.
|
||||||
|
|
||||||
|
\emph{Rolling} versions use the window mean and standard deviation of
|
||||||
|
Section~\ref{sec:estimators} for the annualised return, volatility and Sharpe ratio, plus the
|
||||||
|
cumulative return over the window and the running drawdown; \emph{EWM} versions use the
|
||||||
|
exponentially weighted mean and standard deviation.
|
||||||
|
|
||||||
|
For a portfolio in shares the return series is either the \emph{constant-mix} series
|
||||||
|
\eqref{eq:portret} with today's value weights (the natural benchmark of a rebalanced
|
||||||
|
allocation) or the \emph{buy-and-hold} series $\ln(V_t/V_{t-1})$ with $V_t=\sum_iP_i(t)n_i$
|
||||||
|
(the share numbers held over the whole history). Both are available; the API's
|
||||||
|
\code{/performance/portfolio} defaults to constant mix.
|
||||||
|
|
||||||
|
% ============================================================================================
|
||||||
|
\section{Stock-price models and scenario generation}
|
||||||
|
\label{sec:models}
|
||||||
|
% ============================================================================================
|
||||||
|
|
||||||
|
The optimiser and the scenario-based portfolio risk need a joint distribution of the horizon
|
||||||
|
log-returns of the candidate stocks. Two models are implemented
|
||||||
|
(\code{quantcore.models.simulation}); both assume that one-period log-returns are i.i.d.\ (no
|
||||||
|
autocorrelation), so that an $h$-period return is the sum of $h$ independent one-period draws,
|
||||||
|
both use the \emph{latest} covariance $\Sigma_T$ of a \code{CovarianceModel} (rolling window or
|
||||||
|
EWM, with shrinkage), and both take the expected returns $\mu$ from the expected-return model of
|
||||||
|
Section~\ref{sec:er} (converted to the log-drift $m$ as described in Section~\ref{sec:sampling-mm}).
|
||||||
|
The output is an $S\times N$ matrix of horizon log-returns $r_{s,i}$, optionally with scenario
|
||||||
|
probabilities, together with its provenance (model, covariance model and the shrinkage actually
|
||||||
|
applied, sampling scheme, seed, horizon, date, a fingerprint of the matrix), the \code{ScenarioSet}.
|
||||||
|
|
||||||
|
\subsection{Geometric Brownian motion}
|
||||||
|
|
||||||
|
Under multivariate GBM the horizon log-return is multivariate normal,
|
||||||
|
\begin{equation}
|
||||||
|
r^{(h)} \;\sim\; N\big(h m,\; h\Sigma_T\big),
|
||||||
|
\label{eq:gbm}
|
||||||
|
\end{equation}
|
||||||
|
sampled as $r_s=hm+\sqrt h\,A z_s$ with $z_s\sim N(0,I)$ and $AA^\top=\Sigma_T$ (the
|
||||||
|
eigen-root $V\Lambda^{1/2}$ with eigenvalues descending by default, so that the first coordinate
|
||||||
|
of $z_s$ drives the market factor; the symmetric and Cholesky roots give the same distribution).
|
||||||
|
The draws $z_s$ are scrambled Sobol' points passed through $\Phi^{-1}$ and moment-matched
|
||||||
|
(Section~\ref{sec:sampling-mm}), so mean and covariance of the set are exact. GBM is the natural
|
||||||
|
partner of the Gaussian risk figures of Section~\ref{sec:risk} and of mean-variance thinking; it
|
||||||
|
has no fat tails, no skewness and no tail dependence.
|
||||||
|
|
||||||
|
\subsection{Empirical innovations (filtered historical simulation)}
|
||||||
|
\label{sec:fhs}
|
||||||
|
|
||||||
|
The second model keeps the shape of the observed return distribution and only rescales it to
|
||||||
|
today's covariance. It is the multivariate form of filtered historical simulation
|
||||||
|
\cite{BaroneAdesi1999}.
|
||||||
|
|
||||||
|
\paragraph{Filtering.} For every date $t$ after the warm-up of the estimator, the return vector
|
||||||
|
is standardised with the covariance known at the end of the previous day,
|
||||||
|
\begin{equation}
|
||||||
|
\varepsilon_t \;=\; \Sigma_{t-1}^{-1/2}\, r_t ,
|
||||||
|
\label{eq:innov}
|
||||||
|
\end{equation}
|
||||||
|
where $\Sigma_{t-1}$ comes from the streamed sequence \eqref{eq:ewmrec} (or the rolling
|
||||||
|
recomputation) and $\Sigma^{-1/2}$ is the \emph{symmetric} inverse root $V\Lambda^{-1/2}V^\top$
|
||||||
|
from the eigendecomposition $\Sigma=V\Lambda V^\top$ (eigenvalues floored at $10^{-10}$ of the
|
||||||
|
largest; the Cholesky root is available and used automatically above 250 stocks, where the
|
||||||
|
$O(TN^3)$ eigendecompositions become slow). If the model were exactly right, the innovations would
|
||||||
|
be uncorrelated with unit variance --- and, unlike Gaussian noise, they carry the empirical fat
|
||||||
|
tails, skewness and non-linear co-movements of the data.
|
||||||
|
|
||||||
|
\paragraph{Resampling.} With $A=\Sigma_T^{1/2}$ (symmetric root) and log-drift $m$, the $s$-th
|
||||||
|
horizon scenario draws $h$ dates $j_1,\dots,j_h$ with replacement from the innovation history
|
||||||
|
and sets
|
||||||
|
\begin{equation}
|
||||||
|
r_s \;=\; hm + A\sum_{k=1}^{h}\tilde\varepsilon_{j_k} ,
|
||||||
|
\label{eq:fhs}
|
||||||
|
\end{equation}
|
||||||
|
where $\tilde\varepsilon$ are the centred and re-standardised innovations described next. Since
|
||||||
|
the $h$ draws are independent with identity covariance, $\Cov(r_s)=hA A^\top=h\Sigma_T$,
|
||||||
|
consistent with \eqref{eq:gbm}, while the marginal shape of every asset and the joint tail
|
||||||
|
behaviour come from the data. The dates are drawn by scrambled Sobol' points in $h$ dimensions
|
||||||
|
mapped to the ranks of the innovations sorted by their market-factor component (so that every
|
||||||
|
day of the horizon covers the loss distribution evenly), alternatively uniformly (\code{"mc"})
|
||||||
|
or balanced (every date equally often); the resampled sums are moment-matched to mean zero and
|
||||||
|
covariance $hI$ (Section~\ref{sec:sampling-mm}).
|
||||||
|
|
||||||
|
\paragraph{Centring and re-standardisation.} The innovations are centred so that the scenario
|
||||||
|
mean equals the supplied drift exactly (the drift is the expected-return model's job, not the
|
||||||
|
sample mean's). Their variance, however, is systematically \emph{above one}: any lagged
|
||||||
|
volatility filter is too low right after a shock, so the standardised returns of the following
|
||||||
|
days are too large --- the classical reason why filtered historical simulation rescales the
|
||||||
|
residuals before resampling \cite{BaroneAdesi1999}. Measured on the platform's data set (daily
|
||||||
|
CHF returns 2016--2026, EWM $\lambda=0.99$, automatic shrinkage, symmetric root):
|
||||||
|
|
||||||
|
\begin{center}
|
||||||
|
\small
|
||||||
|
\begin{tabular}{@{}lccc@{}}
|
||||||
|
\toprule
|
||||||
|
number of stocks $N$ & 10 & 30 & 100 \\
|
||||||
|
\midrule
|
||||||
|
median variance of the raw innovations & 1.11 & 1.25 & 1.61 \\
|
||||||
|
scenario volatility / model volatility without re-standardisation & 1.06 & 1.13 & 1.28 \\
|
||||||
|
\bottomrule
|
||||||
|
\end{tabular}
|
||||||
|
\end{center}
|
||||||
|
|
||||||
|
The growth with $N$ is the conditioning effect of Section~\ref{sec:shrinkage}: with an
|
||||||
|
effective window of 100 days and 100 stocks, $\Sigma_{t-1}$ is nearly singular and its inverse
|
||||||
|
root inflates the directions it underestimates. The implementation therefore rescales the
|
||||||
|
centred innovation sample before resampling: \code{standardize="diagonal"} (default) divides
|
||||||
|
each asset's innovations by their sample standard deviation, \code{"full"} whitens them with
|
||||||
|
the inverse root of their sample covariance (only when the history has at least $3N$ dates).
|
||||||
|
After rescaling, the per-asset scenario volatility matches the model volatility within
|
||||||
|
$\pm2\,\%$ (diagonal) or $\pm0.5\,\%$ (full), and the scenario correlations match the model's
|
||||||
|
with a mean absolute error of 0.02 (diagonal) or 0.004 (full). The diagonal option leaves every
|
||||||
|
asset's marginal distribution intact and works for any $N$, which is why it is the default.
|
||||||
|
Finally, innovations beyond $\pm8$ are winsorised as a numerical guard against the same
|
||||||
|
conditioning effect (about 0.2\,\% of the values on the data set); this is a safeguard, not a
|
||||||
|
model choice, and it is reported.
|
||||||
|
|
||||||
|
\subsection{Sampling, moment matching, the drift convention and importance sampling}
|
||||||
|
\label{sec:sampling-mm}
|
||||||
|
|
||||||
|
\paragraph{Why not plain Monte Carlo.} The optimiser is a sample-average approximation: it sees
|
||||||
|
the model only through the scenario set, and the sample mean of a plain Monte-Carlo set of
|
||||||
|
$S$ scenarios is off by $\sigma_i/\sqrt S$ per asset --- for $S=2\,000$ and weekly volatilities
|
||||||
|
of 3--11\,\% that is 3--13\,\% per year, the size of the expected returns themselves. Measured
|
||||||
|
on the 30 largest stocks, two seeds then give portfolios that differ by 100\,\% of wealth and a
|
||||||
|
true expected profit 10\,\% below the optimum. The remedies (all in
|
||||||
|
\code{quantcore.models.sampling}, derived and measured in the optimisation note) are:
|
||||||
|
scrambled Sobol' points instead of pseudo-random draws ($S$ is rounded up to a power of two);
|
||||||
|
\emph{moment matching}, the affine transform $\tilde z_s=C^{-1/2}(z_s-\bar z)$ of the sample
|
||||||
|
that makes its (probability-weighted) mean and covariance exactly $0$ and $I$, so that every
|
||||||
|
scenario set has mean $hm$ and covariance $h\Sigma_T$ exactly and only the shape of the tail is
|
||||||
|
sampled; and the model expected returns in the optimiser's objective instead of the scenario
|
||||||
|
mean. With these, the turnover between two seeds falls to 4--6\,\%, the flatness of the optimum.
|
||||||
|
|
||||||
|
\paragraph{Drift convention.} The expected-return model delivers \emph{arithmetic} expected
|
||||||
|
returns $\mu$ (Section~\ref{sec:er}): reverse optimisation and Black--Litterman are statements
|
||||||
|
about simple returns. The scenarios are generated in log-returns and turned into money with
|
||||||
|
$R=e^{r}-1$, whose expectation under normality is $e^{hm+h\sigma^2/2}-1$. The log-drift is
|
||||||
|
therefore the It\^o-corrected
|
||||||
|
\begin{equation}
|
||||||
|
m_i \;=\; \frac{\ln(1+h\mu_i)}{h}-\frac{\sigma_i^2}{2},
|
||||||
|
\qquad\text{so that}\qquad \E\big[e^{r^{(h)}_i}-1\big]=h\mu_i
|
||||||
|
\label{eq:ito}
|
||||||
|
\end{equation}
|
||||||
|
exactly (\code{drift\_convention="arithmetic"}, the default). Using $\mu_i$ itself as the
|
||||||
|
log-drift (\code{"log"}) adds $\sigma_i^2/2$ to the expected simple return: on the data set the
|
||||||
|
median bonus over the 30 largest stocks is 5.4\,\% per year against a median $\mu$ of 8.9\,\%,
|
||||||
|
and for the most volatile names it is 20--34\,\% (Micron: $\mu=16.9\,\%$ would become
|
||||||
|
$65.6\,\%$) --- a strong and unintended tilt towards volatile stocks. The scenario set reports
|
||||||
|
the expected simple return per asset (\code{expected\_simple}) that the optimiser uses.
|
||||||
|
|
||||||
|
\paragraph{Importance sampling.} Only the $(1-\beta)S$ scenarios in the tail matter for the
|
||||||
|
CVaR. By default (\code{importance\_sampling="auto"}) the sampling distribution is therefore
|
||||||
|
tilted towards the losses of the \emph{cap-weighted portfolio of the candidate stocks}: its loss
|
||||||
|
direction is $d=-A^\top v/\|A^\top v\|$, and two blocks of a quarter of the scenarios each are
|
||||||
|
shifted along $d$ by $z_{0.95}$ and $z_{0.99}$ (GBM: $z_s\mapsto z_s+c_kd$; empirical model: the
|
||||||
|
dates are drawn with exponentially tilted probabilities
|
||||||
|
$q^k_t\propto\exp(\vartheta_k d^\top\tilde\varepsilon_t)$), so that one set covers both usual
|
||||||
|
confidence levels. The shift is a covariance statement: in return space it is
|
||||||
|
$-c_k\Sigma v/\sqrt{v^\top\Sigma v}$, so each asset moves by its covariance with the reference
|
||||||
|
portfolio --- a high-beta stock is stressed hard, a negatively correlated one moves up --- and,
|
||||||
|
an exponential tilt of a Gaussian being a mean shift, the weighted set still has exactly the
|
||||||
|
model covariance and correlations. Every scenario carries the self-normalised likelihood ratio of the mixture
|
||||||
|
proposal as its probability $p_s$, bounded by $1/(1-\sum_k\varepsilon_k)$; the weighted set is
|
||||||
|
moment-matched with these probabilities, the optimiser uses $p_s$ in its CVaR rows and the risk
|
||||||
|
figures of Section~\ref{sec:risk} use the weighted tail. At equal $S$ the tail noise falls by a
|
||||||
|
factor 2--4; at $\beta=0.99$ 2\,048 tilted scenarios are as good as 8\,192 plain ones. Because
|
||||||
|
the direction comes from the market portfolio rather than from a solution of the optimisation,
|
||||||
|
the scenario set does not depend on the risk limit, the constraints or the confidence level ---
|
||||||
|
one set serves a whole family of runs (\code{"adaptive"} tilts towards a first-pass solution
|
||||||
|
instead, which is sharper at $\beta=0.99$ but gives up that property).
|
||||||
|
|
||||||
|
\paragraph{Which model when.} GBM is the transparent baseline; the empirical model is the
|
||||||
|
default of the optimisation because CVaR is a tail measure and the empirical tails are heavier
|
||||||
|
than Gaussian ones (the innovation kurtosis on the data set is well above zero). Both models
|
||||||
|
produce identical second moments by construction, so differences in the optimised portfolios
|
||||||
|
are entirely due to the tail shape.
|
||||||
|
|
||||||
|
\paragraph{Cost.} The filter needs one eigendecomposition per date: for 100 stocks and 2\,500
|
||||||
|
dates about two seconds, for 745 stocks with the Cholesky root and 1\,500 dates about six
|
||||||
|
seconds (four BLAS threads). The innovations depend only on the data and the covariance model,
|
||||||
|
so they can be cached between optimisations with different views or constraints.
|
||||||
|
|
||||||
|
% ============================================================================================
|
||||||
|
\section{The expected-return model}
|
||||||
|
\label{sec:er}
|
||||||
|
% ============================================================================================
|
||||||
|
|
||||||
|
The scenario drift $\mu$ --- and therefore the mean side of the mean--CVaR trade-off --- comes
|
||||||
|
from a Black--Litterman model (\code{quantcore.models.black\_litterman}) on \emph{weekly} CHF
|
||||||
|
log-returns over the last ten years. Weekly data reduce the asynchronous-trading noise of the
|
||||||
|
daily union calendar, and ten years give $\approx520$ observations for the long-term covariance.
|
||||||
|
Everything in this section is per week; annual figures are $52\times$ the means and
|
||||||
|
$\sqrt{52}\times$ the volatilities.
|
||||||
|
|
||||||
|
\subsection{Market capitalisation, weights and the broad-market average}
|
||||||
|
|
||||||
|
The market capitalisation of stock $i$ in CHF is
|
||||||
|
$M_i(t)=\text{shares outstanding}_i\times P_i(t)\times X_{c(i)}(t)$ with today's shares
|
||||||
|
outstanding held constant back in time. The \emph{cap-weighted broad market index} of the whole
|
||||||
|
universe has the log-return
|
||||||
|
\begin{equation}
|
||||||
|
r_M(t) \;=\; \ln\Big(1+\sum_i \omega_i(t-1)\,\big(e^{r_i(t)}-1\big)\Big), \qquad
|
||||||
|
\omega_i(t-1)=\frac{M_i(t-1)}{\sum_jM_j(t-1)},
|
||||||
|
\label{eq:index}
|
||||||
|
\end{equation}
|
||||||
|
i.e.\ \eqref{eq:portret} with weights that drift with prices between rebalancings, exactly like
|
||||||
|
an index. Its average over the ten-year window, converted to an arithmetic expectation,
|
||||||
|
$\mu_M=\overline{r_M}+\tfrac12\widehat{\Var}(r_M)$, is the platform's estimate of the
|
||||||
|
expected market return ("the average of a broad market weighted by current capitalisation");
|
||||||
|
on the data set as of 2026-09-21 the mean log-return is 11.5\,\% per year with a volatility of
|
||||||
|
16.2\,\%, i.e.\ $\mu_M=12.9\,\%$. The current weights $w_i=M_i(T)/\sum_jM_j(T)$, renormalised over the candidate stocks
|
||||||
|
of an optimisation, are the \emph{market weights} of the model.
|
||||||
|
|
||||||
|
\subsection{Reverse optimisation: the equilibrium returns}
|
||||||
|
|
||||||
|
A mean-variance investor with risk aversion $\delta$ solves
|
||||||
|
$\max_w\; w^\top\mu-\frac{\delta}{2}w^\top\Sigma w$, whose first-order condition is
|
||||||
|
$\mu=\delta\Sigma w$. Black and Litterman \cite{BlackLitterman1992,HeLitterman1999} read this
|
||||||
|
backwards: if the market portfolio $w$ is the optimal portfolio of the representative investor,
|
||||||
|
the excess returns that make it optimal are the \emph{implied equilibrium returns}
|
||||||
|
\begin{equation}
|
||||||
|
\pi \;=\; \delta\,\Sigma\,w, \qquad
|
||||||
|
\delta \;=\; \frac{\mu_M - r_f}{w^\top\Sigma w},
|
||||||
|
\label{eq:pi}
|
||||||
|
\end{equation}
|
||||||
|
where $\delta$ is chosen such that the market portfolio itself earns exactly the market excess
|
||||||
|
return: $w^\top\pi=\delta\,w^\top\Sigma w=\mu_M-r_f$. Here $\Sigma$ is the ten-year weekly
|
||||||
|
covariance of the candidate stocks and $r_f$ the weekly CHF risk-free rate (0 by default). The
|
||||||
|
equilibrium returns are proportional to each stock's covariance with the market portfolio ---
|
||||||
|
its beta --- and are the neutral starting point: an investor without views holds the market.
|
||||||
|
On the data set, with the 100 largest stocks as candidates (their market portfolio has a
|
||||||
|
volatility of 19.7\,\%), $\delta\approx3.3$; the equilibrium returns range from about 2.7\,\%
|
||||||
|
(Unilever, Nestl\'e --- low beta in CHF) to about 23\,\% (Nvidia) per year.
|
||||||
|
|
||||||
|
\begin{remark}
|
||||||
|
$\mu_M$ is the average of the \emph{broad} index of all 745 stocks, whereas $w$ and $\Sigma$
|
||||||
|
refer to the candidate subset of an optimisation. This follows the specification (one
|
||||||
|
market-wide average) and keeps $\delta$ stable across candidate sets; the alternative of
|
||||||
|
using the candidate portfolio's own average is a one-line change.
|
||||||
|
\end{remark}
|
||||||
|
|
||||||
|
\subsection{The Black--Litterman posterior}
|
||||||
|
|
||||||
|
The model treats the (unknown) vector of expected excess returns $\mu$ as a random variable with
|
||||||
|
the equilibrium as prior, $\mu\sim N(\pi,\tau\Sigma)$, where the scalar $\tau$ (0.05 by default)
|
||||||
|
expresses how uncertain the equilibrium is relative to the return covariance. $K$ views are
|
||||||
|
linear statements about $\mu$ with their own uncertainty,
|
||||||
|
\begin{equation}
|
||||||
|
P\mu \;=\; Q+\varepsilon, \qquad \varepsilon\sim N(0,\Omega), \quad \Omega=\diag(\omega_1,\dots,\omega_K),
|
||||||
|
\label{eq:views}
|
||||||
|
\end{equation}
|
||||||
|
where each row $p_k$ of $P$ ($K\times N$) is a portfolio: $p_k=e_i$ for an \emph{absolute} view
|
||||||
|
("stock $i$ will return $q_k$"), $p_k=e_i-e_j$ for a \emph{relative} view ("$i$ will outperform
|
||||||
|
$j$ by $q_k$"). Combining prior and views by Bayes' rule (equivalently, generalised least squares)
|
||||||
|
gives the posterior distribution of $\mu$ with mean and covariance
|
||||||
|
\begin{equation}
|
||||||
|
\mu_{\mathrm{BL}} = \big[(\tau\Sigma)^{-1}+P^\top\Omega^{-1}P\big]^{-1}\big[(\tau\Sigma)^{-1}\pi+P^\top\Omega^{-1}Q\big], \qquad
|
||||||
|
\Sigma_\mu = \big[(\tau\Sigma)^{-1}+P^\top\Omega^{-1}P\big]^{-1}.
|
||||||
|
\label{eq:bl1}
|
||||||
|
\end{equation}
|
||||||
|
The implementation uses the algebraically identical form that needs no inverse of $\tau\Sigma$
|
||||||
|
(a $K\times K$ system only), which is numerically preferable and also valid when $\Sigma$ is
|
||||||
|
singular:
|
||||||
|
\begin{equation}
|
||||||
|
\mu_{\mathrm{BL}} = \pi + \tau\Sigma P^\top\big(P\tau\Sigma P^\top+\Omega\big)^{-1}(Q-P\pi), \qquad
|
||||||
|
\Sigma_{\mathrm{BL}} = \Sigma + \tau\Sigma - \tau\Sigma P^\top\big(P\tau\Sigma P^\top+\Omega\big)^{-1}P\tau\Sigma .
|
||||||
|
\label{eq:bl2}
|
||||||
|
\end{equation}
|
||||||
|
The posterior mean is the prior corrected in the direction of the views by the amount by which
|
||||||
|
they disagree with the prior ($Q-P\pi$), weighted by how confident they are relative to the
|
||||||
|
prior; $\Sigma_{\mathrm{BL}}=\Sigma+\Sigma_\mu$ is the covariance of the returns themselves
|
||||||
|
(return noise plus the remaining uncertainty about the mean) and is reported for completeness.
|
||||||
|
Without views, $\mu_{\mathrm{BL}}=\pi$. The posterior mean is the drift handed to the scenario
|
||||||
|
models: per day, $\mu=\mu_{\mathrm{BL}}^{\mathrm{ann}}/252$ (total return, i.e.\ excess plus
|
||||||
|
$r_f$).
|
||||||
|
|
||||||
|
\subsection{Idzorek's confidence: from a degree of belief to $\Omega$}
|
||||||
|
|
||||||
|
The weak point of the model is $\Omega$: nobody can state the variance of a view. Idzorek
|
||||||
|
\cite{Idzorek2005} replaces it by a \emph{confidence} $C_k\in(0,1]$ with a portfolio meaning.
|
||||||
|
Consider one view $k$ alone. At 100\,\% confidence ($\omega_k=0$) the posterior is
|
||||||
|
$\mu_{100}=\pi+\tau\Sigma p_k^\top(p_k\tau\Sigma p_k^\top)^{-1}(q_k-p_k\pi)$ and the
|
||||||
|
unconstrained mean-variance weights move from $w$ to $w_{100}=(\delta\Sigma)^{-1}\mu_{100}$.
|
||||||
|
Idzorek defines the confidence as the fraction of that move the investor wants to make:
|
||||||
|
the target weights are $w+C_k\,(w_{100}-w)$, and $\omega_k$ is the value for which the
|
||||||
|
posterior weights $w(\omega_k)=(\delta\Sigma)^{-1}\mu_{\mathrm{BL}}(\omega_k)$ hit the target
|
||||||
|
(in the paper by a one-dimensional least-squares search). For a single view this calibration
|
||||||
|
has a closed form. From \eqref{eq:bl2} with one view,
|
||||||
|
\begin{equation}
|
||||||
|
w(\omega_k)-w \;=\; (\delta\Sigma)^{-1}\tau\Sigma p_k^\top\,\frac{q_k-p_k\pi}{p_k\tau\Sigma p_k^\top+\omega_k}
|
||||||
|
\;=\; \frac{\tau}{\delta}\,p_k^\top\,\frac{q_k-p_k\pi}{p_k\tau\Sigma p_k^\top+\omega_k},
|
||||||
|
\end{equation}
|
||||||
|
so the tilt is always in the direction $p_k^\top$ and its size relative to the full tilt
|
||||||
|
($\omega_k=0$) is $p_k\tau\Sigma p_k^\top/(p_k\tau\Sigma p_k^\top+\omega_k)$. Setting this equal to
|
||||||
|
$C_k$ gives
|
||||||
|
\begin{equation}
|
||||||
|
\boxed{\;\omega_k \;=\; \frac{1-C_k}{C_k}\;p_k\,\tau\Sigma\,p_k^\top\;}
|
||||||
|
\label{eq:idzorek}
|
||||||
|
\end{equation}
|
||||||
|
\cite{Walters2014}: the view's variance is the prior variance of the view portfolio scaled by
|
||||||
|
the odds against the view; $C_k=1$ gives $\omega_k=0$ (the view is imposed), $C_k=0.5$ gives
|
||||||
|
$\omega_k=p_k\tau\Sigma p_k^\top$ (the He--Litterman choice, available as
|
||||||
|
\code{omega\_method="he\_litterman"}), $C_k\to0$ removes the view. The implementation uses
|
||||||
|
\eqref{eq:idzorek} for every view and then computes the joint posterior \eqref{eq:bl2}; the
|
||||||
|
numerical search of the paper is kept as \code{idzorek\_omega\_numeric} and a unit test shows
|
||||||
|
that both agree to $10^{-4}$. With several views, each $\omega_k$ is calibrated for its view
|
||||||
|
alone, as in the paper.
|
||||||
|
|
||||||
|
\begin{example}[Four assets]
|
||||||
|
\label{ex:bl}
|
||||||
|
Annual volatilities $(20,25,15,30)\,\%$, correlations
|
||||||
|
$\rho_{12}=0.5,\rho_{13}=0.3,\rho_{14}=0.2,\rho_{23}=0.4,\rho_{24}=0.1,\rho_{34}=0.3$, market
|
||||||
|
weights $(0.4,0.3,0.2,0.1)$, market return 7\,\%, $r_f=1\,\%$, $\tau=0.05$, weekly
|
||||||
|
covariance $\Sigma=\Sigma^{\mathrm{ann}}/52$. Then $w^\top\Sigma^{\mathrm{ann}}w=0.0250$
|
||||||
|
(market volatility 15.8\,\%), $\delta=0.06/0.0250=2.40$, and the equilibrium total returns
|
||||||
|
are $\pi+r_f=(7.36,\ 8.80,\ 4.35,\ 5.50)\,\%$ --- the market portfolio earns
|
||||||
|
$0.4\cdot7.36+0.3\cdot8.80+0.2\cdot4.35+0.1\cdot5.50=7.00\,\%$. An absolute view "asset 2
|
||||||
|
returns 15\,\%" with confidence $C=0.4$ gives $\omega=\frac{0.6}{0.4}\,p\tau\Sigma p^\top=1.5\times
|
||||||
|
6.01\cdot10^{-5}=9.01\cdot10^{-5}$ (weekly units; the numerical search returns the same value)
|
||||||
|
and the posterior total returns $(8.35,\ 11.28,\ 4.94,\ 5.80)\,\%$: asset 2 moves 40\,\% of the
|
||||||
|
way from 8.80\,\% to the 15\,\% it would reach at full confidence, and the correlated assets
|
||||||
|
follow. The unconstrained weights move from $0.3$ to $0.4656$ for asset 2 --- exactly
|
||||||
|
$0.3+0.4\,(0.7139-0.3)$ --- and stay unchanged for the others, as an absolute view on one
|
||||||
|
asset only tilts that asset. The numbers are pinned by \code{tests/test\_black\_litterman.py}.
|
||||||
|
\end{example}
|
||||||
|
|
||||||
|
\subsection{Views from the AI layer: sentiment with a degree of belief}
|
||||||
|
|
||||||
|
The platform's news pipeline delivers, per stock, an assessment "positive / negative" with a
|
||||||
|
degree of belief rather than a return forecast. \code{view\_from\_sentiment} turns
|
||||||
|
$(\text{stock } i,\ \text{sentiment } s\in[-1,1],\ \text{confidence } C)$ into the absolute view
|
||||||
|
\begin{equation}
|
||||||
|
q \;=\; \pi_i^{\mathrm{ann}} + r_f^{\mathrm{ann}} + s\cdot\gamma\cdot\sigma_i^{\mathrm{ann}}, \qquad \gamma=0.25 \text{ by default},
|
||||||
|
\label{eq:sentiment}
|
||||||
|
\end{equation}
|
||||||
|
i.e.\ a fully positive assessment claims one quarter of the stock's annual volatility above its
|
||||||
|
equilibrium return (about $+5\,\%$ for a 20\,\%-volatility stock), and the degree of belief
|
||||||
|
becomes the Idzorek confidence $C$. The strength $\gamma$ is a modelling parameter; the view
|
||||||
|
carries the provenance of the insight that produced it (\code{source}). On the data set, a
|
||||||
|
sentiment of $+0.8$ with confidence $0.6$ on Nestl\'e states $q=6.1\,\%$ against an equilibrium
|
||||||
|
of $2.7\,\%$, calibrates $\omega=1.8\cdot10^{-5}$ and lifts the posterior to $4.7\,\%$ per
|
||||||
|
year. Explicit views (\code{assets}, \code{q}, \code{confidence}), including relative ones, are
|
||||||
|
accepted alongside.
|
||||||
|
|
||||||
|
\subsection{Units and alternatives}
|
||||||
|
|
||||||
|
Views state \emph{annual total} returns; internally they are converted to weekly excess returns
|
||||||
|
($q/52$ minus the weekly $r_f$ times the sum of the view weights, which vanishes for relative
|
||||||
|
views). Results are reported per week and annualised; the optimiser receives the posterior
|
||||||
|
total return per day, $\mu_{\mathrm{BL}}^{\mathrm{ann}}/252$, as the \emph{arithmetic}
|
||||||
|
expected return, converts it to the It\^o-corrected log-drift \eqref{eq:ito} of \eqref{eq:gbm}
|
||||||
|
and \eqref{eq:fhs}, and uses the expected simple return over the horizon,
|
||||||
|
$h\mu_{\mathrm{BL}}^{\mathrm{ann}}/252$, in its objective. Inputs that are log-return means
|
||||||
|
(the index average, the historical means) are converted with $+\tfrac12\sigma^2$. Two simpler
|
||||||
|
models are available for comparison: \code{expected\_returns="historical"} (the ten-year weekly
|
||||||
|
means, converted) and \code{"zero"} (no drift, pure risk minimisation).
|
||||||
|
|
||||||
|
% ============================================================================================
|
||||||
|
\section{Implementation map}
|
||||||
|
% ============================================================================================
|
||||||
|
|
||||||
|
\begin{table}[h]
|
||||||
|
\centering
|
||||||
|
\footnotesize
|
||||||
|
\begin{tabular}{@{}p{3.2cm}p{5.6cm}p{6.4cm}@{}}
|
||||||
|
\toprule
|
||||||
|
concept & code & parameters (defaults) / output \\
|
||||||
|
\midrule
|
||||||
|
log-returns, alignment, frequencies, horizons & \code{measurement.returns}: \code{log\_returns}, \code{align\_prices}, \code{resample\_prices}, \code{horizon\_periods}, \code{horizon\_returns}, \code{portfolio\_log\_returns} & \code{freq} daily $|$ weekly $|$ monthly; \code{horizon} day $|$ week $|$ month $|$ integer \\
|
||||||
|
CHF conversion & \code{data.fx.FXConverter}; \newline \code{MarketDataService.prices}, \code{.returns} & \code{currency="base"} (CHF) $|$ \code{"local"} \\
|
||||||
|
rolling / EWM series & \code{estimators.rolling\_mean}, \code{rolling\_vol}, \code{ewm\_mean}, \code{ewm\_vol} & \code{window=200}, \code{lam=0.94}; annualised with the periods per year \\
|
||||||
|
covariance matrices & \code{estimators.rolling\_cov}, \code{ewm\_cov}, \code{covariance}, \code{cov\_to\_corr}, \code{covariance\_sequence}; \newline \code{models.covariance.CovarianceModel} & \code{method} rolling $|$ ewm, \code{window}, \code{lam}, \code{shrinkage} $\in[0,1]$ or \code{"auto"} \\
|
||||||
|
VaR / CVaR & \code{risk.normal\_var}, \code{normal\_cvar}, \code{historical\_var}, \code{historical\_cvar}, \code{scenario\_cvar}, \code{loss\_to\_money} & $\beta\in\{0.95,0.99\}$; losses positive; money via $V_0(1-e^{-\ell})$ \\
|
||||||
|
per-asset risk table & \code{RiskService.asset\_risk} $\to$ \code{asset\_risk\_table} & columns \code{mean\_rolling\_ann}, \code{vol\_rolling\_ann}, \code{vol\_ewm\_ann}, \code{var95\_normal\_rolling}, \code{cvar95\_normal\_rolling}, \code{var95\_normal\_ewm}, \code{cvar95\_normal\_ewm}, \code{var95\_hist}, \code{cvar95\_hist} (and the same for 99), \code{horizon\_periods}, \code{n\_obs}, \code{as\_of} \\
|
||||||
|
risk time series & \code{RiskService.risk\_series} & same names, one row per date \\
|
||||||
|
portfolio risk & \code{RiskService.portfolio\_risk} $\to$ \code{PortfolioRisk} & \code{holdings} (shares), \code{cash}, \code{method} normal $|$ historical $|$ scenarios, \code{cov\_method}; per level \code{var}, \code{cvar}, \code{var\_money}, \code{cvar\_money} \\
|
||||||
|
performance & \code{performance\_table}, \code{rolling\_performance}, \code{ewm\_performance}, \code{PerformanceService} & \code{risk\_free} from the settings; \code{rebalanced=True} = constant mix \\
|
||||||
|
scenario models & \code{models.simulation.GBMSimulator}, \newline \code{EmpiricalInnovationSimulator} $\to$ \code{ScenarioSet} & \code{n\_scenarios=4096}, \code{seed=42}, \code{sampling} sobol $|$ mc $|$ antithetic $|$ lhs $|$ balanced, \code{moment\_match=True}, \code{tilt} (\code{TiltSpec}: direction, shifts, mixtures), \code{drift\_convention} arithmetic $|$ log, \code{root}, \code{standardize} diagonal $|$ full $|$ None, \code{clip=8}, \code{max\_history=2500}; output \code{probabilities}, \code{expected\_simple}, \code{hash} \\
|
||||||
|
sampling schemes & \code{models.sampling}: \code{standard\_normal\_sample}, \code{moment\_match}, \code{pca\_root}, \code{tilt\_normal\_sample}, \code{tilted\_resample}, \code{resample\_indices}, \code{scenario\_hash} & Sobol' sizes are powers of two; likelihood ratios bounded by $1/(1-\sum_k\code{mixture}_k)$ \\
|
||||||
|
one set per family of runs & \code{OptimizationService.scenario\_key}, \code{.market\_exposure}, \code{.clear\_scenario\_cache}; \code{ScenarioSet.with\_drift}, \code{.shocks}, \code{.shock\_hash} & the key excludes risk limit, confidence, constraints, costs, objective, cash, holdings and the expected-return model \\
|
||||||
|
market index and weights & \code{MarketDataService}: \newline \code{cap\_weighted\_index}, \code{market\_caps}, \code{market\_weights} & shares outstanding $\times$ price $\times$ FX \\
|
||||||
|
Black--Litterman & \code{BlackLittermanModel}, \code{View}, \code{view\_from\_sentiment}, \code{idzorek\_omega} $\to$ \code{BlackLittermanResult} & \code{tau=0.05}, \code{periods\_per\_year=52}, \code{omega\_method} idzorek $|$ he\_litterman; \code{prior\_ann}, \code{posterior\_ann}, \code{omega}, \code{delta}, \code{market\_return\_ann} \\
|
||||||
|
pipeline in the optimiser & \code{OptimizationService}: \newline \code{expected\_returns}, \code{scenarios} & \code{bl\_years=10}, \code{expected\_returns} black\_litterman $|$ historical $|$ zero, \code{views}, \code{covariance} = \{ewm, 0.99, auto\}, \code{scenario\_model} empirical $|$ gbm $|$ gaussian\_exact, \code{importance\_sampling} auto $|$ market $|$ adaptive $|$ off, \code{row\_generation} auto $|$ on $|$ off \\
|
||||||
|
HTTP & \code{/risk/*}, \code{/performance/*}, \code{/model/expected-returns}, \code{/model/scenarios}, \code{/market/*} & see \code{/docs} \\
|
||||||
|
\bottomrule
|
||||||
|
\end{tabular}
|
||||||
|
\caption{Where each concept lives in \code{quantcore}.}
|
||||||
|
\label{tab:impl}
|
||||||
|
\end{table}
|
||||||
|
|
||||||
|
\paragraph{Determinism and provenance.} Every figure is a deterministic function of the stored
|
||||||
|
data and the stated parameters; simulations take an explicit seed. Every result carries the
|
||||||
|
parameters that produced it (window, $\lambda$, shrinkage applied, model, seed, horizon, date,
|
||||||
|
views with their calibrated $\omega$), so that the advisor layer can cite them and a reviewer
|
||||||
|
can recompute them.
|
||||||
|
|
||||||
|
% ============================================================================================
|
||||||
|
\begin{thebibliography}{99}
|
||||||
|
\bibitem{RiskMetrics1996}
|
||||||
|
J.P.~Morgan/Reuters, \emph{RiskMetrics --- Technical Document}, 4th edition, New York, 1996.
|
||||||
|
\bibitem{LedoitWolf2004}
|
||||||
|
O.~Ledoit and M.~Wolf, \emph{A well-conditioned estimator for large-dimensional covariance
|
||||||
|
matrices}, Journal of Multivariate Analysis 88(2), 365--411, 2004.
|
||||||
|
\bibitem{SchaeferStrimmer2005}
|
||||||
|
J.~Sch\"afer and K.~Strimmer, \emph{A shrinkage approach to large-scale covariance matrix
|
||||||
|
estimation and implications for functional genomics}, Statistical Applications in Genetics and
|
||||||
|
Molecular Biology 4(1), Article 32, 2005.
|
||||||
|
\bibitem{AcerbiTasche2002}
|
||||||
|
C.~Acerbi and D.~Tasche, \emph{On the coherence of expected shortfall},
|
||||||
|
Journal of Banking \& Finance 26(7), 1487--1503, 2002.
|
||||||
|
\bibitem{MFE2015}
|
||||||
|
A.\,J.~McNeil, R.~Frey and P.~Embrechts, \emph{Quantitative Risk Management: Concepts,
|
||||||
|
Techniques and Tools}, revised edition, Princeton University Press, 2015.
|
||||||
|
\bibitem{BaroneAdesi1999}
|
||||||
|
G.~Barone-Adesi, K.~Giannopoulos and L.~Vosper, \emph{VaR without correlations for portfolios
|
||||||
|
of derivative securities}, Journal of Futures Markets 19(5), 583--602, 1999.
|
||||||
|
\bibitem{BlackLitterman1992}
|
||||||
|
F.~Black and R.~Litterman, \emph{Global portfolio optimization},
|
||||||
|
Financial Analysts Journal 48(5), 28--43, 1992.
|
||||||
|
\bibitem{HeLitterman1999}
|
||||||
|
G.~He and R.~Litterman, \emph{The intuition behind Black--Litterman model portfolios},
|
||||||
|
Goldman Sachs Investment Management Research, 1999.
|
||||||
|
\bibitem{Idzorek2005}
|
||||||
|
T.\,M.~Idzorek, \emph{A step-by-step guide to the Black--Litterman model: incorporating
|
||||||
|
user-specified confidence levels}, working paper, Ibbotson Associates, 2005 (also in
|
||||||
|
S.~Satchell (ed.), \emph{Forecasting Expected Returns in the Financial Markets}, 2007).
|
||||||
|
\bibitem{Walters2014}
|
||||||
|
J.~Walters, \emph{The Black--Litterman model in detail}, working paper (SSRN), 2014.
|
||||||
|
\end{thebibliography}
|
||||||
|
|
||||||
|
\end{document}
|
||||||
BIN
docs/optimization_model.pdf
Normal file
BIN
docs/optimization_model.pdf
Normal file
Binary file not shown.
1197
docs/optimization_model.tex
Normal file
1197
docs/optimization_model.tex
Normal file
File diff suppressed because it is too large
Load Diff
49
pyproject.toml
Normal file
49
pyproject.toml
Normal file
@ -0,0 +1,49 @@
|
|||||||
|
[build-system]
|
||||||
|
requires = ["setuptools>=68", "wheel"]
|
||||||
|
build-backend = "setuptools.build_meta"
|
||||||
|
|
||||||
|
[project]
|
||||||
|
name = "quantcore"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "Deterministic quant core of the AI-Augmented Portfolio Intelligence Platform: data, risk/performance measurement, stock-price models, Black-Litterman expected returns and mean-CVaR MILP optimisation on NVIDIA cuOpt."
|
||||||
|
readme = "README.md"
|
||||||
|
requires-python = ">=3.11"
|
||||||
|
license = { text = "MIT" }
|
||||||
|
authors = [{ name = "Florian Herzog" }]
|
||||||
|
dependencies = [
|
||||||
|
"numpy>=1.26",
|
||||||
|
"pandas>=2.1",
|
||||||
|
"scipy>=1.11",
|
||||||
|
"pyarrow>=14",
|
||||||
|
"yfinance>=0.2.50",
|
||||||
|
"requests>=2.31",
|
||||||
|
"lxml>=4.9",
|
||||||
|
"pydantic>=2.5",
|
||||||
|
"pydantic-settings>=2.1",
|
||||||
|
"fastapi>=0.109",
|
||||||
|
"uvicorn[standard]>=0.27",
|
||||||
|
"msgpack>=1.0",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
dev = ["pytest>=7.4", "pytest-cov", "httpx>=0.25", "ruff"]
|
||||||
|
|
||||||
|
[project.scripts]
|
||||||
|
quantcore = "quantcore.cli:main"
|
||||||
|
quantcore-api = "quantcore.apicli:main"
|
||||||
|
|
||||||
|
[tool.setuptools.packages.find]
|
||||||
|
include = ["quantcore*"]
|
||||||
|
|
||||||
|
[tool.setuptools.package-data]
|
||||||
|
"quantcore.data" = ["static/*.csv", "static/*.json"]
|
||||||
|
"quantcore.api" = ["ui/*.html", "ui/*.css", "ui/js/*.js", "ui/js/tabs/*.js"]
|
||||||
|
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
testpaths = ["tests"]
|
||||||
|
addopts = "-q"
|
||||||
|
filterwarnings = ["ignore::DeprecationWarning", "ignore::FutureWarning"]
|
||||||
|
|
||||||
|
[tool.ruff]
|
||||||
|
line-length = 110
|
||||||
|
target-version = "py311"
|
||||||
24
quantcore/__init__.py
Normal file
24
quantcore/__init__.py
Normal file
@ -0,0 +1,24 @@
|
|||||||
|
"""quantcore -- the deterministic quant core of the AI-Augmented Portfolio Intelligence Platform.
|
||||||
|
|
||||||
|
The package is organised as a set of *services* with plain-Python contracts:
|
||||||
|
|
||||||
|
* :mod:`quantcore.universe` -- ``SetupService``: builds the investment universe once
|
||||||
|
(largest 500 US + 500 European stocks, metadata, first price date, market capitalisation).
|
||||||
|
* :mod:`quantcore.data` -- ``DataDownloadService`` and ``DataStore``: Yahoo Finance
|
||||||
|
prices and FX rates, incremental updates, snapshots, resilient HTTP access.
|
||||||
|
* :mod:`quantcore.measurement` -- ``RiskService`` / ``PerformanceService``: log-returns,
|
||||||
|
moving-window and exponentially weighted volatility, correlation, VaR and CVaR, performance.
|
||||||
|
* :mod:`quantcore.models` -- covariance estimators, market index, stock-price simulation
|
||||||
|
(GBM and empirical innovations), Black-Litterman with Idzorek confidence views.
|
||||||
|
* :mod:`quantcore.optimization` -- mean-CVaR MILP in integer share numbers solved on NVIDIA
|
||||||
|
cuOpt (with a local HiGHS fallback), in "new portfolio" and "rebalancing" mode.
|
||||||
|
* :mod:`quantcore.api` -- FastAPI application exposing the services over HTTP/JSON.
|
||||||
|
|
||||||
|
Everything in here is deterministic: same input, same output. No LLM is called anywhere in
|
||||||
|
this package -- the AI layer of the platform consumes these services through their contracts.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from quantcore.config import Settings, get_settings
|
||||||
|
|
||||||
|
__all__ = ["Settings", "get_settings", "__version__"]
|
||||||
|
__version__ = "0.1.0"
|
||||||
1
quantcore/api/__init__.py
Normal file
1
quantcore/api/__init__.py
Normal file
@ -0,0 +1 @@
|
|||||||
|
"""HTTP/JSON API (FastAPI) exposing the deterministic services."""
|
||||||
556
quantcore/api/app.py
Normal file
556
quantcore/api/app.py
Normal file
@ -0,0 +1,556 @@
|
|||||||
|
"""FastAPI application -- the HTTP/JSON surface of the quant core.
|
||||||
|
|
||||||
|
Run with ``quantcore serve`` (or ``uvicorn quantcore.api.app:app``). Interactive docs at
|
||||||
|
``/docs``. Long-running work (setup, data update) runs as background jobs; everything else
|
||||||
|
answers synchronously. All numbers come from the deterministic services -- the API adds no
|
||||||
|
logic of its own.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import threading
|
||||||
|
import uuid
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
from fastapi import BackgroundTasks, FastAPI, HTTPException, Query
|
||||||
|
from fastapi.responses import RedirectResponse
|
||||||
|
from fastapi.staticfiles import StaticFiles
|
||||||
|
|
||||||
|
from quantcore import __version__
|
||||||
|
from quantcore.api import schemas as S
|
||||||
|
from quantcore.config import Settings, get_settings
|
||||||
|
from quantcore.data.download_service import DataDownloadService
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.data.store import DataStore
|
||||||
|
from quantcore.measurement.performance import PerformanceService, ewm_performance, rolling_performance
|
||||||
|
from quantcore.measurement.returns import horizon_periods, periods_per_year
|
||||||
|
from quantcore.measurement.risk import RiskService
|
||||||
|
from quantcore.models.covariance import CovarianceModel
|
||||||
|
from quantcore.optimization.service import OptimizationRequest, OptimizationService
|
||||||
|
from quantcore.optimization.solvers import CuOptSolver
|
||||||
|
from quantcore.universe.setup_service import SetupService
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
app = FastAPI(
|
||||||
|
title="quantcore -- Portfolio Risk Measurement & Optimisation",
|
||||||
|
version=__version__,
|
||||||
|
description=(
|
||||||
|
"Deterministic services of the AI-Augmented Portfolio Intelligence Platform: market data, "
|
||||||
|
"risk and performance measurement, stock-price models, Black-Litterman expected returns and "
|
||||||
|
"mean-CVaR MILP optimisation (NVIDIA cuOpt)."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# state
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
class State:
|
||||||
|
def __init__(self, settings: Settings | None = None):
|
||||||
|
self.settings = settings or get_settings()
|
||||||
|
self.store = DataStore(self.settings)
|
||||||
|
self._md: MarketDataService | None = None
|
||||||
|
self.jobs: dict[str, dict] = {}
|
||||||
|
self.lock = threading.Lock()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def md(self) -> MarketDataService:
|
||||||
|
if self._md is None:
|
||||||
|
if not self.store.has_prices():
|
||||||
|
raise HTTPException(503, "no market data yet - run the setup job (POST /jobs/setup)")
|
||||||
|
self._md = MarketDataService(self.settings, self.store)
|
||||||
|
return self._md
|
||||||
|
|
||||||
|
def reload(self) -> None:
|
||||||
|
self._md = None
|
||||||
|
|
||||||
|
|
||||||
|
state = State()
|
||||||
|
|
||||||
|
|
||||||
|
def _records(df: pd.DataFrame, index_name: str = "symbol") -> list[dict]:
|
||||||
|
out = df.copy()
|
||||||
|
out.index.name = index_name
|
||||||
|
out = out.reset_index()
|
||||||
|
return json.loads(out.to_json(orient="records", date_format="iso", double_precision=10))
|
||||||
|
|
||||||
|
|
||||||
|
def _frame(df: pd.DataFrame) -> dict:
|
||||||
|
"""Wide frame -> ``{"index": [...], "columns": [...], "data": [[...], ...]}`` (NaN -> null)."""
|
||||||
|
return {
|
||||||
|
"index": [i.isoformat() if hasattr(i, "isoformat") else str(i) for i in df.index],
|
||||||
|
"columns": [str(c) for c in df.columns],
|
||||||
|
"data": json.loads(json.dumps(df.values.tolist(), default=float)) if df.size else [],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _clean(obj: Any) -> Any:
|
||||||
|
"""Recursively turn numpy / NaN / timestamps into JSON-safe values."""
|
||||||
|
if isinstance(obj, dict):
|
||||||
|
return {str(k): _clean(v) for k, v in obj.items()}
|
||||||
|
if isinstance(obj, (list, tuple)):
|
||||||
|
return [_clean(v) for v in obj]
|
||||||
|
if isinstance(obj, (np.floating, float)):
|
||||||
|
return None if not np.isfinite(obj) else float(obj)
|
||||||
|
if isinstance(obj, (np.integer,)):
|
||||||
|
return int(obj)
|
||||||
|
if isinstance(obj, np.ndarray):
|
||||||
|
return _clean(obj.tolist())
|
||||||
|
if isinstance(obj, (pd.Timestamp, datetime)):
|
||||||
|
return obj.isoformat()
|
||||||
|
if isinstance(obj, pd.Series):
|
||||||
|
return _clean(obj.to_dict())
|
||||||
|
return obj
|
||||||
|
|
||||||
|
|
||||||
|
def _http(exc: Exception) -> HTTPException:
|
||||||
|
if isinstance(exc, (KeyError, ValueError)):
|
||||||
|
return HTTPException(422, str(exc))
|
||||||
|
if isinstance(exc, FileNotFoundError):
|
||||||
|
return HTTPException(503, str(exc))
|
||||||
|
log.exception("request failed")
|
||||||
|
return HTTPException(500, f"{type(exc).__name__}: {exc}")
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# health, universe, data
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
@app.get("/health", tags=["system"])
|
||||||
|
def health() -> dict:
|
||||||
|
return _clean({"status": "ok", "version": __version__, "store": state.store.status(),
|
||||||
|
"cuopt": CuOptSolver(state.settings).health(), "base_currency": state.settings.base_currency})
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/universe", tags=["data"])
|
||||||
|
def universe(region: str | None = None, sector: str | None = None, country: str | None = None) -> list[dict]:
|
||||||
|
try:
|
||||||
|
md = state.md
|
||||||
|
symbols = md.symbols(region, sector, country)
|
||||||
|
return _records(md.universe.loc[symbols])
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/universe/{symbol}", tags=["data"])
|
||||||
|
def universe_symbol(symbol: str) -> dict:
|
||||||
|
md = state.md
|
||||||
|
if symbol not in md.universe.index:
|
||||||
|
raise HTTPException(404, f"{symbol} not in the universe")
|
||||||
|
return _clean({"symbol": symbol, **md.universe.loc[symbol].to_dict()})
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/prices", tags=["data"])
|
||||||
|
def prices(symbols: str = Query(..., description="comma-separated symbols"), start: str | None = None,
|
||||||
|
end: str | None = None, currency: str = "base", freq: str = "daily") -> dict:
|
||||||
|
try:
|
||||||
|
px = state.md.prices(symbols.split(","), start, end, currency, freq)
|
||||||
|
return {"currency": currency, "freq": freq, **_frame(px)}
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/returns", tags=["data"])
|
||||||
|
def returns(symbols: str = Query(...), start: str | None = None, end: str | None = None, currency: str = "base",
|
||||||
|
freq: str = "daily") -> dict:
|
||||||
|
try:
|
||||||
|
r = state.md.returns(symbols.split(","), start, end, currency, freq)
|
||||||
|
return {"currency": currency, "freq": freq, "kind": "log_returns", **_frame(r)}
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/data/symbols", tags=["data"])
|
||||||
|
def data_symbols(region: str | None = None, sector: str | None = None, country: str | None = None,
|
||||||
|
search: str | None = None, limit: int = Query(200, ge=1, le=2000),
|
||||||
|
sort: str = Query("market_cap", pattern="^(market_cap|symbol|name|sector)$")) -> dict:
|
||||||
|
"""The universe as one table (metadata, current market cap and price) plus the filter facets.
|
||||||
|
|
||||||
|
One call feeds a stock picker: the matching rows and the counts per region, sector and country.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
md = state.md
|
||||||
|
u = md.universe
|
||||||
|
caps = md.market_caps().reindex(u.index)
|
||||||
|
px = md.latest_prices(u.index.tolist(), currency="base").reindex(u.index)
|
||||||
|
table = pd.DataFrame({
|
||||||
|
"name": u["name"], "region": u["region"], "country": u["country"], "sector": u["gics_sector"],
|
||||||
|
"industry": u["yahoo_industry"], "currency": u["listing_currency"], "exchange": u["exchange"],
|
||||||
|
"market_cap": caps, "market_weight": caps / caps.sum(), "price": px,
|
||||||
|
"price_local": u["last_price"], "first_price_date": u["first_price_date"].astype(str),
|
||||||
|
"price_coverage": u["price_coverage"], "index_weight": u["index_weight"],
|
||||||
|
})
|
||||||
|
facets = {"regions": table["region"].value_counts().to_dict(),
|
||||||
|
"sectors": table["sector"].fillna("Unknown").value_counts().to_dict(),
|
||||||
|
"countries": table["country"].value_counts().to_dict()}
|
||||||
|
m = table
|
||||||
|
if region:
|
||||||
|
m = m[m["region"].str.lower() == region.lower()]
|
||||||
|
if sector:
|
||||||
|
m = m[m["sector"].fillna("Unknown").str.lower() == sector.lower()]
|
||||||
|
if country:
|
||||||
|
m = m[m["country"].str.lower() == country.lower()]
|
||||||
|
if search:
|
||||||
|
q = search.lower()
|
||||||
|
m = m[[q in str(i).lower() or q in str(n).lower() for i, n in zip(m.index, m["name"])]]
|
||||||
|
m = m.sort_values("market_cap", ascending=False) if sort == "market_cap" else m.sort_values(sort)
|
||||||
|
return _clean({"as_of": md.last_date(), "base_currency": state.settings.base_currency,
|
||||||
|
"n_universe": int(len(table)), "n_matched": int(len(m)), "facets": facets,
|
||||||
|
"symbols": _records(m.head(limit))})
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/market/index", tags=["data"])
|
||||||
|
def market_index(start: str | None = None, end: str | None = None, freq: str = "weekly") -> dict:
|
||||||
|
try:
|
||||||
|
r = state.md.cap_weighted_index(start=start, end=end, freq=freq)
|
||||||
|
ppy = periods_per_year(freq)
|
||||||
|
return _clean({"freq": freq, "mean_ann": float(r.mean() * ppy), "vol_ann": float(r.std() * np.sqrt(ppy)),
|
||||||
|
"n": int(len(r)), **_frame(r.to_frame())})
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/market/weights", tags=["data"])
|
||||||
|
def market_weights(max_assets: int | None = None) -> dict:
|
||||||
|
try:
|
||||||
|
w = state.md.market_weights().sort_values(ascending=False)
|
||||||
|
if max_assets:
|
||||||
|
w = w.iloc[:max_assets]
|
||||||
|
return _clean({"as_of": state.md.last_date(), "weights": w.round(8).to_dict()})
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# risk
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
@app.post("/risk/assets", tags=["risk"])
|
||||||
|
def risk_assets(req: S.AssetRiskRequest) -> list[dict]:
|
||||||
|
try:
|
||||||
|
table = RiskService(state.md).asset_risk(req.symbols, req.freq, req.currency, req.window, req.lam,
|
||||||
|
tuple(req.confidence), req.horizon, req.start, req.end,
|
||||||
|
space=req.space, drift=req.drift)
|
||||||
|
return _records(table)
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/risk/series", tags=["risk"])
|
||||||
|
def risk_series(req: S.RiskSeriesRequest) -> dict:
|
||||||
|
try:
|
||||||
|
df = RiskService(state.md).risk_series(req.symbol, req.freq, req.currency, req.window, req.lam,
|
||||||
|
tuple(req.confidence), req.horizon, req.start, req.end)
|
||||||
|
return {"symbol": req.symbol, **_frame(df.dropna(how="all"))}
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/risk/covariance", tags=["risk"])
|
||||||
|
def risk_covariance(req: S.CovarianceRequest) -> dict:
|
||||||
|
try:
|
||||||
|
cov = RiskService(state.md).covariance(req.symbols, req.freq, req.currency, req.method, req.window, req.lam,
|
||||||
|
req.shrinkage, req.start, req.end)
|
||||||
|
return {"method": req.method, "freq": req.freq, **_frame(cov)}
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/risk/correlation", tags=["risk"])
|
||||||
|
def risk_correlation(req: S.CovarianceRequest) -> dict:
|
||||||
|
try:
|
||||||
|
corr = RiskService(state.md).correlation(req.symbols, req.freq, req.currency, req.method, req.window, req.lam,
|
||||||
|
req.start, req.end)
|
||||||
|
return {"method": req.method, "freq": req.freq, **_frame(corr)}
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/risk/portfolio", tags=["risk"])
|
||||||
|
def risk_portfolio(req: S.PortfolioRiskRequest) -> dict:
|
||||||
|
try:
|
||||||
|
pr = RiskService(state.md).portfolio_risk(req.holdings, req.cash, req.freq, req.currency, req.horizon,
|
||||||
|
tuple(req.confidence), req.method, req.cov_method, req.window,
|
||||||
|
req.lam, req.shrinkage, req.start, req.end)
|
||||||
|
return _clean(pr.as_dict())
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# performance
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
def _perf() -> PerformanceService:
|
||||||
|
return PerformanceService(state.md, state.settings.risk_free_rate)
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/performance/assets", tags=["performance"])
|
||||||
|
def performance_assets(req: S.PerformanceRequest) -> list[dict]:
|
||||||
|
try:
|
||||||
|
return _records(_perf().asset_performance(req.symbols, req.freq, req.currency, req.start, req.end))
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/performance/rolling", tags=["performance"])
|
||||||
|
def performance_rolling(req: S.RollingPerformanceRequest) -> dict:
|
||||||
|
try:
|
||||||
|
r = state.md.returns([req.symbol], req.start, req.end, req.currency, req.freq)[req.symbol].dropna()
|
||||||
|
rf = state.settings.risk_free_rate
|
||||||
|
df = ewm_performance(r, req.lam, req.freq, rf) if req.lam else rolling_performance(r, req.window, req.freq, rf)
|
||||||
|
return {"symbol": req.symbol, **_frame(df.dropna(how="all"))}
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/performance/portfolio", tags=["performance"])
|
||||||
|
def performance_portfolio(req: S.PortfolioPerformanceRequest) -> dict:
|
||||||
|
try:
|
||||||
|
row = _perf().portfolio_performance(req.holdings, req.freq, req.currency, req.start, req.end, req.rebalanced)
|
||||||
|
return _clean(row.to_dict())
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# models
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
@app.post("/model/expected-returns", tags=["models"])
|
||||||
|
def expected_returns(req: S.ExpectedReturnsRequest) -> dict:
|
||||||
|
try:
|
||||||
|
md = state.md
|
||||||
|
svc = OptimizationService(state.settings, md)
|
||||||
|
oreq = OptimizationRequest(symbols=req.symbols, max_assets=req.max_assets, views=[v.as_dict() for v in req.views],
|
||||||
|
bl_tau=req.tau, bl_years=req.years, bl_omega_method=req.omega_method, cvar_limit=1.0)
|
||||||
|
symbols = svc.candidate_symbols(oreq)
|
||||||
|
# the very model the optimiser uses (arithmetic equilibrium return, same tau, same views)
|
||||||
|
mu_ann, desc, result = svc.expected_returns(oreq, symbols)
|
||||||
|
start = (md.last_date() - pd.DateOffset(years=req.years)).date().isoformat()
|
||||||
|
weekly = md.returns(symbols, start=start, currency="base", freq="weekly").dropna()
|
||||||
|
out = result.as_dict()
|
||||||
|
out["volatility_ann"] = (weekly.std() * np.sqrt(52)).round(6).to_dict()
|
||||||
|
out["expected_returns_ann"] = mu_ann.round(6).to_dict()
|
||||||
|
out["model"] = desc
|
||||||
|
out["omega_method"] = req.omega_method
|
||||||
|
return _clean(out)
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/model/covariance", tags=["models"])
|
||||||
|
def model_covariance(req: S.ModelCovarianceRequest) -> dict:
|
||||||
|
"""Covariance and correlation of the estimator the scenario generator and the optimiser use.
|
||||||
|
|
||||||
|
Unlike ``/risk/covariance`` (a measurement with explicit parameters) this takes the same
|
||||||
|
``covariance`` model dict as ``/optimize``, scales it to the horizon and reports how well
|
||||||
|
conditioned the estimate is for the number of assets.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
md = state.md
|
||||||
|
model = CovarianceModel.from_dict(req.covariance)
|
||||||
|
r = md.returns(req.symbols, start=req.start, end=req.end, currency="base", freq=req.freq).dropna()
|
||||||
|
cov = model.estimate(r)
|
||||||
|
h = horizon_periods(req.horizon, req.freq)
|
||||||
|
ppy = periods_per_year(req.freq)
|
||||||
|
sd = np.sqrt(np.diag(cov.values))
|
||||||
|
corr = pd.DataFrame(cov.values / np.outer(sd, sd), index=cov.index, columns=cov.columns)
|
||||||
|
vals = np.linalg.eigvalsh(cov.values)[::-1]
|
||||||
|
off = corr.values[np.triu_indices(len(sd), 1)]
|
||||||
|
return _clean({
|
||||||
|
"model": model.describe(), "shrinkage_applied": model.intensity(r),
|
||||||
|
"conditioning": model.conditioning(r), "observations": int(len(r)),
|
||||||
|
"freq": req.freq, "horizon_periods": h, "as_of": r.index[-1],
|
||||||
|
"volatility_ann": dict(zip(cov.index, sd * np.sqrt(ppy))),
|
||||||
|
"volatility_horizon": dict(zip(cov.index, sd * np.sqrt(h))),
|
||||||
|
"eigenvalues": (vals / vals.sum()).tolist(),
|
||||||
|
"correlation_summary": {"mean": float(off.mean()), "min": float(off.min()), "max": float(off.max())},
|
||||||
|
"covariance": _frame(cov * h), "correlation": _frame(corr),
|
||||||
|
})
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/model/scenarios", tags=["models"])
|
||||||
|
def scenarios(req: S.ScenarioRequest) -> dict:
|
||||||
|
try:
|
||||||
|
md = state.md
|
||||||
|
svc = OptimizationService(state.settings, md)
|
||||||
|
oreq = OptimizationRequest(symbols=req.symbols, expected_returns=req.expected_returns, cvar_limit=1.0,
|
||||||
|
horizon=req.horizon, freq=req.freq, covariance=req.covariance, seed=req.seed,
|
||||||
|
n_scenarios=req.n_scenarios, scenario_model=req.model,
|
||||||
|
innovation_history=req.innovation_history, sampling=req.sampling,
|
||||||
|
moment_match=req.moment_match, drift_convention=req.drift_convention,
|
||||||
|
importance_sampling=req.importance_sampling, bl_tau=req.bl_tau,
|
||||||
|
bl_years=req.bl_years, bl_omega_method=req.bl_omega_method,
|
||||||
|
views=[v.as_dict() for v in req.views])
|
||||||
|
symbols = svc.candidate_symbols(oreq)
|
||||||
|
mu_ann, desc, _ = svc.expected_returns(oreq, symbols)
|
||||||
|
scen = svc.scenarios(oreq, symbols, mu_ann)
|
||||||
|
simple = scen.simple_returns()
|
||||||
|
out = {"description": scen.describe(), "expected_returns": desc, "scenario_key": scen.extra.get("key"),
|
||||||
|
"expected_returns_ann": _clean(mu_ann.reindex(symbols).to_dict()),
|
||||||
|
"summary": _records(scen.summary()), "horizon_periods": horizon_periods(req.horizon, req.freq)}
|
||||||
|
if req.focus:
|
||||||
|
out["asset_histograms"] = {}
|
||||||
|
for sym in req.focus:
|
||||||
|
if sym not in symbols:
|
||||||
|
continue
|
||||||
|
col = simple[:, symbols.index(sym)]
|
||||||
|
mass, edges = np.histogram(col, bins=req.bins, weights=scen.weights)
|
||||||
|
out["asset_histograms"][sym] = {"edges": edges.tolist(), "mass": mass.tolist()}
|
||||||
|
exposure, value = _exposure(req, symbols)
|
||||||
|
if exposure is not None:
|
||||||
|
out["portfolio"] = {
|
||||||
|
"value": value, "cash": req.cash, "invested": float(exposure.sum()),
|
||||||
|
"exposure": {s: float(v) for s, v in zip(symbols, exposure) if v != 0},
|
||||||
|
**scen.distribution(exposure, bins=req.bins, confidence=tuple(req.confidence)),
|
||||||
|
}
|
||||||
|
return _clean(out)
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _exposure(req: S.ScenarioRequest, symbols: list[str]) -> tuple[np.ndarray | None, float]:
|
||||||
|
"""Money exposure per candidate from ``holdings`` (shares) or ``weights`` (of ``value``)."""
|
||||||
|
if req.holdings:
|
||||||
|
px = state.md.latest_prices(symbols, currency="base").reindex(symbols)
|
||||||
|
v = np.array([float(req.holdings.get(s, 0.0)) for s in symbols]) * px.values
|
||||||
|
return v, float(v.sum() + req.cash)
|
||||||
|
if req.weights:
|
||||||
|
v = np.array([float(req.weights.get(s, 0.0)) for s in symbols]) * req.value
|
||||||
|
return v, float(req.value)
|
||||||
|
return None, 0.0
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# optimisation
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
@app.post("/optimize", tags=["optimisation"])
|
||||||
|
def optimize(req: S.OptimizeRequest) -> dict:
|
||||||
|
try:
|
||||||
|
payload = req.model_dump()
|
||||||
|
payload["views"] = [v.as_dict() for v in req.views]
|
||||||
|
oreq = OptimizationRequest(**payload)
|
||||||
|
result = OptimizationService(state.settings, state.md).optimize(oreq)
|
||||||
|
return _clean(result.as_dict())
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
raise _http(exc) from exc
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# jobs: setup and data update (background), snapshots
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
def _run_job(job_id: str, fn) -> None:
|
||||||
|
job = state.jobs[job_id]
|
||||||
|
job["status"], job["started_at"] = "running", datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||||
|
try:
|
||||||
|
job["result"] = _clean(fn())
|
||||||
|
job["status"] = "done"
|
||||||
|
state.reload()
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
log.exception("job %s failed", job_id)
|
||||||
|
job["status"], job["error"] = "failed", f"{type(exc).__name__}: {exc}"
|
||||||
|
job["finished_at"] = datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||||
|
try:
|
||||||
|
state.settings.jobs_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
(state.settings.jobs_dir / f"{job_id}.json").write_text(json.dumps(job, indent=2, default=str))
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _submit(kind: str, fn, background: BackgroundTasks) -> dict:
|
||||||
|
with state.lock:
|
||||||
|
running = [j for j in state.jobs.values() if j["status"] in ("queued", "running") and j["kind"] in ("setup", "update")]
|
||||||
|
if running and kind in ("setup", "update"):
|
||||||
|
raise HTTPException(409, f"a {running[0]['kind']} job is already running ({running[0]['id']})")
|
||||||
|
job_id = uuid.uuid4().hex[:12]
|
||||||
|
state.jobs[job_id] = {"id": job_id, "kind": kind, "status": "queued", "started_at": None, "finished_at": None,
|
||||||
|
"result": None, "error": None}
|
||||||
|
background.add_task(_run_job, job_id, fn)
|
||||||
|
return state.jobs[job_id]
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/jobs/setup", tags=["jobs"], status_code=202)
|
||||||
|
def job_setup(background: BackgroundTasks, offline: bool = False, max_per_region: int | None = None) -> dict:
|
||||||
|
return _submit("setup", lambda: SetupService(state.settings, state.store).run(allow_web=not offline,
|
||||||
|
max_per_region=max_per_region).as_dict(),
|
||||||
|
background)
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/jobs/update", tags=["jobs"], status_code=202)
|
||||||
|
def job_update(background: BackgroundTasks, full: bool = False) -> dict:
|
||||||
|
return _submit("update", lambda: DataDownloadService(state.settings, state.store).update(full=full).as_dict(), background)
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/jobs", tags=["jobs"])
|
||||||
|
def jobs() -> list[dict]:
|
||||||
|
return list(state.jobs.values())
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/jobs/{job_id}", tags=["jobs"])
|
||||||
|
def job(job_id: str) -> dict:
|
||||||
|
if job_id not in state.jobs:
|
||||||
|
raise HTTPException(404, "unknown job")
|
||||||
|
return state.jobs[job_id]
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/snapshots", tags=["jobs"], status_code=201)
|
||||||
|
def create_snapshot(req: S.SnapshotRequest) -> dict:
|
||||||
|
try:
|
||||||
|
return {"path": str(state.store.create_snapshot(req.name))}
|
||||||
|
except FileExistsError as exc:
|
||||||
|
raise HTTPException(409, f"snapshot exists: {exc}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/snapshots", tags=["jobs"])
|
||||||
|
def list_snapshots() -> list[str]:
|
||||||
|
return state.store.list_snapshots()
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# the test bench (single-page UI served from this app)
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
UI_DIR = Path(__file__).with_name("ui")
|
||||||
|
|
||||||
|
|
||||||
|
class _NoCacheStatic(StaticFiles):
|
||||||
|
"""Static files that must be revalidated on every request.
|
||||||
|
|
||||||
|
The bench is edited while it is open, and without a ``Cache-Control`` header a browser applies
|
||||||
|
heuristic caching to the ES modules: it keeps serving yesterday's ``optimize.js`` and the reader
|
||||||
|
sees a fixed bug that is still there. ``no-cache`` does not mean "do not store" -- the ETag still
|
||||||
|
produces a 304 for an unchanged file, so revalidation costs a round trip and no body.
|
||||||
|
"""
|
||||||
|
|
||||||
|
async def get_response(self, path: str, scope): # noqa: ANN001 (Starlette's own signature)
|
||||||
|
response = await super().get_response(path, scope)
|
||||||
|
response.headers["Cache-Control"] = "no-cache, must-revalidate"
|
||||||
|
return response
|
||||||
|
|
||||||
|
|
||||||
|
def mount_test_bench(application: FastAPI, settings: Settings) -> bool:
|
||||||
|
"""Serve the static test bench at ``/ui/`` and redirect ``/`` to it; ``True`` when mounted.
|
||||||
|
|
||||||
|
A deployment-time switch (``QUANTCORE_ENABLE_UI=false``, read once when this module is
|
||||||
|
imported): the bench is a client of this API, not part of it, so an installation whose users
|
||||||
|
build their own front end can ship the same image without it. ``/docs`` is unaffected.
|
||||||
|
"""
|
||||||
|
if not settings.enable_ui or not UI_DIR.is_dir():
|
||||||
|
return False
|
||||||
|
application.mount("/ui", _NoCacheStatic(directory=UI_DIR, html=True), name="ui")
|
||||||
|
|
||||||
|
@application.get("/", include_in_schema=False)
|
||||||
|
def _root() -> RedirectResponse:
|
||||||
|
return RedirectResponse("/ui/")
|
||||||
|
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
UI_ENABLED = mount_test_bench(app, state.settings)
|
||||||
233
quantcore/api/schemas.py
Normal file
233
quantcore/api/schemas.py
Normal file
@ -0,0 +1,233 @@
|
|||||||
|
"""Pydantic request/response models of the HTTP API (the *contracts* of the services)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
|
||||||
|
class SymbolsRequest(BaseModel):
|
||||||
|
symbols: list[str] = Field(..., min_length=1, description="Yahoo symbols from the universe")
|
||||||
|
freq: str = Field("daily", pattern="^(daily|weekly|monthly)$")
|
||||||
|
currency: str = Field("base", pattern="^(base|local)$")
|
||||||
|
start: str | None = None
|
||||||
|
end: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class AssetRiskRequest(SymbolsRequest):
|
||||||
|
window: int = Field(200, ge=10, description="moving-window length in periods")
|
||||||
|
lam: float = Field(0.94, gt=0, lt=1, description="EWM decay")
|
||||||
|
confidence: list[float] = Field(default=[0.95, 0.99])
|
||||||
|
horizon: str | int = Field("day", description="'day' | 'week' | 'month' | integer periods")
|
||||||
|
drift: str = Field("estimated", pattern="^(estimated|zero)$",
|
||||||
|
description="'estimated': the window's mean enters the parametric VaR/CVaR; 'zero': "
|
||||||
|
"volatility only (the usual market-risk convention, and the only sane one "
|
||||||
|
"for long horizons, where the mean is the noisiest input)")
|
||||||
|
space: str = Field("log", pattern="^(log|value)$",
|
||||||
|
description="'log': VaR/CVaR as log-returns (the modelling space); 'value': as a "
|
||||||
|
"fraction of the position's value, 1 - exp(-x), which cannot reach 100 %")
|
||||||
|
|
||||||
|
|
||||||
|
class RiskSeriesRequest(BaseModel):
|
||||||
|
symbol: str
|
||||||
|
freq: str = Field("daily", pattern="^(daily|weekly|monthly)$")
|
||||||
|
currency: str = Field("base", pattern="^(base|local)$")
|
||||||
|
window: int = Field(200, ge=10)
|
||||||
|
lam: float = Field(0.94, gt=0, lt=1)
|
||||||
|
confidence: list[float] = Field(default=[0.95, 0.99])
|
||||||
|
horizon: str | int = "day"
|
||||||
|
start: str | None = None
|
||||||
|
end: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class CovarianceRequest(SymbolsRequest):
|
||||||
|
method: str = Field("rolling", pattern="^(rolling|ewm)$")
|
||||||
|
window: int = Field(200, ge=10)
|
||||||
|
lam: float = Field(0.94, gt=0, lt=1)
|
||||||
|
shrinkage: float = Field(0.0, ge=0, le=1)
|
||||||
|
|
||||||
|
|
||||||
|
class PortfolioRiskRequest(BaseModel):
|
||||||
|
holdings: dict[str, float] = Field(..., description="number of shares per symbol")
|
||||||
|
cash: float = 0.0
|
||||||
|
freq: str = Field("daily", pattern="^(daily|weekly|monthly)$")
|
||||||
|
currency: str = Field("base", pattern="^(base|local)$")
|
||||||
|
horizon: str | int = "week"
|
||||||
|
confidence: list[float] = Field(default=[0.95, 0.99])
|
||||||
|
method: str = Field("normal", pattern="^(normal|historical)$")
|
||||||
|
cov_method: str = Field("rolling", pattern="^(rolling|ewm)$")
|
||||||
|
window: int = Field(200, ge=10)
|
||||||
|
lam: float = Field(0.94, gt=0, lt=1)
|
||||||
|
shrinkage: float = Field(0.0, ge=0, le=1)
|
||||||
|
start: str | None = None
|
||||||
|
end: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class PerformanceRequest(SymbolsRequest):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class RollingPerformanceRequest(BaseModel):
|
||||||
|
symbol: str
|
||||||
|
window: int = Field(252, ge=10)
|
||||||
|
lam: float | None = Field(None, gt=0, lt=1, description="if given: exponentially weighted instead of rolling")
|
||||||
|
freq: str = Field("daily", pattern="^(daily|weekly|monthly)$")
|
||||||
|
currency: str = Field("base", pattern="^(base|local)$")
|
||||||
|
start: str | None = None
|
||||||
|
end: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class PortfolioPerformanceRequest(BaseModel):
|
||||||
|
holdings: dict[str, float]
|
||||||
|
freq: str = Field("daily", pattern="^(daily|weekly|monthly)$")
|
||||||
|
currency: str = Field("base", pattern="^(base|local)$")
|
||||||
|
start: str | None = None
|
||||||
|
end: str | None = None
|
||||||
|
rebalanced: bool = True
|
||||||
|
|
||||||
|
|
||||||
|
class ViewSpec(BaseModel):
|
||||||
|
"""Either an explicit view (``assets`` + ``q``) or a sentiment view (``symbol`` + ``sentiment``)."""
|
||||||
|
|
||||||
|
assets: dict[str, float] | None = None
|
||||||
|
q: float | None = Field(None, description="annual expected return of the view portfolio")
|
||||||
|
symbol: str | None = None
|
||||||
|
sentiment: float | None = Field(None, ge=-1, le=1)
|
||||||
|
strength: float = Field(0.25, ge=0)
|
||||||
|
confidence: float = Field(0.5, gt=0, le=1)
|
||||||
|
label: str = ""
|
||||||
|
source: str = ""
|
||||||
|
|
||||||
|
def as_dict(self) -> dict:
|
||||||
|
d = self.model_dump(exclude_none=True)
|
||||||
|
return d
|
||||||
|
|
||||||
|
|
||||||
|
class ExpectedReturnsRequest(BaseModel):
|
||||||
|
symbols: list[str] | None = None
|
||||||
|
max_assets: int = 100
|
||||||
|
views: list[ViewSpec] = Field(default_factory=list)
|
||||||
|
tau: float = Field(0.05, gt=0)
|
||||||
|
years: int = Field(10, ge=1)
|
||||||
|
omega_method: str = Field("idzorek", pattern="^(idzorek|he_litterman)$")
|
||||||
|
|
||||||
|
|
||||||
|
class ModelCovarianceRequest(BaseModel):
|
||||||
|
"""Covariance as the *models* see it (the estimator of the scenario generator / optimiser)."""
|
||||||
|
|
||||||
|
symbols: list[str] = Field(..., min_length=1)
|
||||||
|
freq: str = Field("daily", pattern="^(daily|weekly)$")
|
||||||
|
covariance: dict = Field(default_factory=lambda: {"method": "ewm", "lam": 0.99, "shrinkage": "auto"})
|
||||||
|
horizon: str | int = "week"
|
||||||
|
start: str | None = None
|
||||||
|
end: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class ScenarioRequest(BaseModel):
|
||||||
|
symbols: list[str] = Field(..., min_length=1)
|
||||||
|
model: str = Field("empirical", pattern="^(gbm|empirical)$")
|
||||||
|
n_scenarios: int = Field(4096, ge=100, le=200_000, description="rounded up to a power of two for Sobol' sampling")
|
||||||
|
horizon: str | int = "week"
|
||||||
|
freq: str = Field("daily", pattern="^(daily|weekly)$")
|
||||||
|
covariance: dict = Field(default_factory=lambda: {"method": "ewm", "lam": 0.99, "shrinkage": "auto"})
|
||||||
|
seed: int = 42
|
||||||
|
sampling: str = Field("sobol", pattern="^(sobol|mc|antithetic|lhs|balanced)$")
|
||||||
|
moment_match: bool = Field(True, description="exact scenario mean and covariance")
|
||||||
|
drift_convention: str = Field("arithmetic", pattern="^(arithmetic|log)$")
|
||||||
|
expected_returns: str = Field("zero", pattern="^(zero|historical|black_litterman)$")
|
||||||
|
innovation_history: int | None = 2500
|
||||||
|
importance_sampling: str = Field("auto", pattern="^(auto|market|adaptive|off)$")
|
||||||
|
views: list["ViewSpec"] = Field(default_factory=list)
|
||||||
|
bl_tau: float = Field(0.05, gt=0)
|
||||||
|
bl_years: int = Field(10, ge=1)
|
||||||
|
bl_omega_method: str = Field("idzorek", pattern="^(idzorek|he_litterman)$")
|
||||||
|
# the portfolio whose P&L distribution is reported (shares, or weights of `value`)
|
||||||
|
holdings: dict[str, float] = Field(default_factory=dict)
|
||||||
|
weights: dict[str, float] = Field(default_factory=dict)
|
||||||
|
value: float = Field(1_000_000.0, gt=0, description="portfolio value when `weights` are given")
|
||||||
|
cash: float = Field(0.0, ge=0)
|
||||||
|
confidence: list[float] = Field(default=[0.95, 0.99])
|
||||||
|
bins: int = Field(60, ge=5, le=400)
|
||||||
|
focus: list[str] = Field(default_factory=list, description="assets to return a histogram for")
|
||||||
|
|
||||||
|
|
||||||
|
class ConstraintsSpec(BaseModel):
|
||||||
|
max_weight: float = Field(0.10, gt=0, le=1)
|
||||||
|
max_sector_weight: float | dict[str, float] = 0.25
|
||||||
|
max_positions: int | None = Field(None, ge=1)
|
||||||
|
min_position_value: float = Field(0.0, ge=0)
|
||||||
|
max_turnover: float | None = Field(None, ge=0)
|
||||||
|
max_transaction_cost: float | None = Field(None, ge=0)
|
||||||
|
min_invested: float = Field(0.0, ge=0, le=1)
|
||||||
|
max_invested: float = Field(1.0, gt=0, le=1)
|
||||||
|
|
||||||
|
|
||||||
|
class CostsSpec(BaseModel):
|
||||||
|
proportional_buy: float = Field(0.001, ge=0)
|
||||||
|
proportional_sell: float = Field(0.001, ge=0)
|
||||||
|
fixed_per_trade: float = Field(0.0, ge=0)
|
||||||
|
|
||||||
|
|
||||||
|
class OptimizeRequest(BaseModel):
|
||||||
|
mode: str = Field("new", pattern="^(new|rebalance)$")
|
||||||
|
cash: float = Field(1_000_000.0, ge=0)
|
||||||
|
holdings: dict[str, int] = Field(default_factory=dict)
|
||||||
|
symbols: list[str] | None = None
|
||||||
|
max_assets: int = Field(100, ge=2)
|
||||||
|
horizon: str | int = "week"
|
||||||
|
freq: str = Field("daily", pattern="^(daily|weekly)$")
|
||||||
|
confidence: float = Field(0.95, gt=0.5, lt=1)
|
||||||
|
cvar_limit: float | None = Field(None, gt=0, description="max expected shortfall in CHF over the horizon")
|
||||||
|
cvar_limit_fraction: float | None = Field(None, gt=0, lt=1)
|
||||||
|
objective: str = Field("max_return", pattern="^(max_return|min_cvar|utility)$")
|
||||||
|
risk_aversion: float = Field(1.0, ge=0)
|
||||||
|
target_return: float | None = None
|
||||||
|
scenario_model: str = Field("empirical", pattern="^(gbm|empirical|gaussian_exact)$",
|
||||||
|
description="'gaussian_exact': no scenarios, outer approximation of the normal model")
|
||||||
|
n_scenarios: int | None = Field(None, ge=100, le=200_000,
|
||||||
|
description="default: 16 per candidate, within [4096, 16384]; rounded up to a "
|
||||||
|
"power of two for Sobol' sampling. About 64 per candidate is "
|
||||||
|
"needed to pin the composition; the result warns below 8")
|
||||||
|
seed: int = 42
|
||||||
|
sampling: str = Field("sobol", pattern="^(sobol|mc|antithetic|lhs|balanced)$")
|
||||||
|
moment_match: bool = Field(True, description="exact scenario mean and covariance")
|
||||||
|
importance_sampling: str = Field("auto", pattern="^(auto|market|adaptive|off)$",
|
||||||
|
description="'market': tilt towards the tail of the cap-weighted candidate "
|
||||||
|
"portfolio (independent of the risk parameters, so a family of runs "
|
||||||
|
"shares one scenario set); 'adaptive': tilt towards a first-pass "
|
||||||
|
"solution; 'auto': 'market' when the tail would be thin")
|
||||||
|
is_mixture: float = Field(0.25, gt=0, lt=0.5, description="share of scenarios per tilted block")
|
||||||
|
drift_convention: str = Field("arithmetic", pattern="^(arithmetic|log)$")
|
||||||
|
risk_drift: str = Field("model", pattern="^(model|zero)$",
|
||||||
|
description="'model': the CVaR is measured on the scenarios including the expected "
|
||||||
|
"return; 'zero': on the centred scenarios, so a high return forecast "
|
||||||
|
"cannot relax the risk limit (the usual market-risk convention)")
|
||||||
|
row_generation: str = Field("auto", pattern="^(auto|on|off)$",
|
||||||
|
description="solve on the tail rows only (large scenario sets)")
|
||||||
|
covariance: dict = Field(default_factory=lambda: {"method": "ewm", "lam": 0.99, "shrinkage": "auto"})
|
||||||
|
expected_returns: str = Field("black_litterman", pattern="^(black_litterman|historical|zero)$")
|
||||||
|
views: list[ViewSpec] = Field(default_factory=list)
|
||||||
|
bl_tau: float = Field(0.05, gt=0)
|
||||||
|
bl_years: int = Field(10, ge=1)
|
||||||
|
bl_omega_method: str = Field("idzorek", pattern="^(idzorek|he_litterman)$")
|
||||||
|
constraints: ConstraintsSpec = Field(default_factory=ConstraintsSpec)
|
||||||
|
costs: CostsSpec = Field(default_factory=CostsSpec)
|
||||||
|
solver: str = Field("auto", pattern="^(auto|cuopt|highs)$")
|
||||||
|
time_limit: float | None = Field(None, gt=0)
|
||||||
|
innovation_history: int | None = 2500
|
||||||
|
innovation_standardize: str | None = Field("diagonal", pattern="^(diagonal|full)$")
|
||||||
|
|
||||||
|
|
||||||
|
class JobStatus(BaseModel):
|
||||||
|
id: str
|
||||||
|
kind: str
|
||||||
|
status: str # queued | running | done | failed
|
||||||
|
started_at: str | None = None
|
||||||
|
finished_at: str | None = None
|
||||||
|
result: Any | None = None
|
||||||
|
error: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class SnapshotRequest(BaseModel):
|
||||||
|
name: str | None = None
|
||||||
27
quantcore/api/ui/index.html
Normal file
27
quantcore/api/ui/index.html
Normal file
@ -0,0 +1,27 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>quantcore test bench</title>
|
||||||
|
<meta name="description" content="Test bench for the quantcore portfolio risk and optimisation services.">
|
||||||
|
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'><rect width='16' height='16' rx='3' fill='%232a78d6'/><path d='M3 11.5 L6.2 6.5 L9 9 L13 3.7' stroke='white' stroke-width='1.7' fill='none' stroke-linecap='round' stroke-linejoin='round'/></svg>">
|
||||||
|
<link rel="stylesheet" href="styles.css">
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div id="app" class="app">
|
||||||
|
<header class="topbar">
|
||||||
|
<div class="brand"><b>quantcore</b><span>test bench</span></div>
|
||||||
|
<div class="status-row" id="status"><span class="pill"><span class="dot"></span>checking …</span></div>
|
||||||
|
<button class="btn small" id="theme" type="button" title="light / dark / follow the system">Theme</button>
|
||||||
|
</header>
|
||||||
|
<nav class="tabs" id="tabs" role="tablist"></nav>
|
||||||
|
<div class="body">
|
||||||
|
<aside class="sidebar" id="sidebar"></aside>
|
||||||
|
<main class="main" id="main"></main>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div id="tooltip" role="status" aria-live="polite"></div>
|
||||||
|
<script type="module" src="js/main.js"></script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
508
quantcore/api/ui/js/charts.js
Normal file
508
quantcore/api/ui/js/charts.js
Normal file
@ -0,0 +1,508 @@
|
|||||||
|
/* Small SVG chart library: line, bar, histogram, heatmap, scatter.
|
||||||
|
No dependencies (the container may have no internet). Colours come from the CSS custom
|
||||||
|
properties in styles.css, so the theme switch is a re-render, never a second palette.
|
||||||
|
Every chart keeps its config on the node and re-draws on resize and on a theme change. */
|
||||||
|
|
||||||
|
const NS = "http://www.w3.org/2000/svg";
|
||||||
|
const PLOT = { left: 58, right: 16, top: 10, bottom: 26 };
|
||||||
|
|
||||||
|
export function svgEl(name, attrs = {}) {
|
||||||
|
const e = document.createElementNS(NS, name);
|
||||||
|
for (const [k, v] of Object.entries(attrs)) if (v !== null && v !== undefined) e.setAttribute(k, String(v));
|
||||||
|
return e;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function cssVar(name) {
|
||||||
|
return getComputedStyle(document.documentElement).getPropertyValue(name).trim() || "#888";
|
||||||
|
}
|
||||||
|
|
||||||
|
export function seriesColor(i) { return cssVar(`--series-${(i % 8) + 1}`); }
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ scales and ticks */
|
||||||
|
const lin = (d0, d1, r0, r1) => (v) => (d1 === d0 ? (r0 + r1) / 2 : r0 + ((v - d0) / (d1 - d0)) * (r1 - r0));
|
||||||
|
|
||||||
|
export function niceTicks(lo, hi, count = 5) {
|
||||||
|
if (!isFinite(lo) || !isFinite(hi)) return [0];
|
||||||
|
if (lo === hi) return [lo];
|
||||||
|
const span = (hi - lo) / count;
|
||||||
|
const mag = Math.pow(10, Math.floor(Math.log10(Math.abs(span))));
|
||||||
|
const step = [1, 2, 2.5, 5, 10].map((m) => m * mag).find((s) => s >= span) || 10 * mag;
|
||||||
|
const out = [];
|
||||||
|
for (let t = Math.ceil(lo / step) * step; t <= hi + step * 1e-6; t += step) out.push(Math.abs(t) < step * 1e-9 ? 0 : t);
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
function padDomain([lo, hi], pad = 0.06, includeZero = false) {
|
||||||
|
if (includeZero) { lo = Math.min(lo, 0); hi = Math.max(hi, 0); }
|
||||||
|
if (lo === hi) { const d = Math.abs(lo) || 1; return [lo - d * 0.1, hi + d * 0.1]; }
|
||||||
|
const p = (hi - lo) * pad;
|
||||||
|
return [lo - p, hi + p];
|
||||||
|
}
|
||||||
|
|
||||||
|
const extent = (vals) => vals.reduce(([a, b], v) => [Math.min(a, v), Math.max(b, v)], [Infinity, -Infinity]);
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ tooltip (singleton) */
|
||||||
|
function tipNode() {
|
||||||
|
let t = document.getElementById("tooltip");
|
||||||
|
if (!t) { t = document.createElement("div"); t.id = "tooltip"; document.body.appendChild(t); }
|
||||||
|
return t;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function showTip(ev, title, rows) {
|
||||||
|
const t = tipNode();
|
||||||
|
t.replaceChildren();
|
||||||
|
if (title) { const h = document.createElement("div"); h.className = "tt-title"; h.textContent = title; t.appendChild(h); }
|
||||||
|
for (const r of rows) {
|
||||||
|
const row = document.createElement("div"); row.className = "tt-row";
|
||||||
|
if (r.color) { const k = document.createElement("span"); k.className = "key"; k.style.background = r.color; row.appendChild(k); }
|
||||||
|
const nm = document.createElement("span"); nm.className = "nm"; nm.textContent = r.name ?? ""; row.appendChild(nm);
|
||||||
|
const v = document.createElement("span"); v.className = "v"; v.textContent = r.value ?? ""; row.appendChild(v);
|
||||||
|
t.appendChild(row);
|
||||||
|
}
|
||||||
|
t.style.opacity = "1";
|
||||||
|
const pad = 14, w = t.offsetWidth, h = t.offsetHeight;
|
||||||
|
let x = ev.clientX + pad, y = ev.clientY + pad;
|
||||||
|
if (x + w > window.innerWidth - 8) x = ev.clientX - w - pad;
|
||||||
|
if (y + h > window.innerHeight - 8) y = ev.clientY - h - pad;
|
||||||
|
t.style.left = `${Math.max(4, x)}px`; t.style.top = `${Math.max(4, y)}px`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function hideTip() { const t = document.getElementById("tooltip"); if (t) t.style.opacity = "0"; }
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ chart frame */
|
||||||
|
function frame(node, cfg) {
|
||||||
|
const height = cfg.height || 220;
|
||||||
|
const margin = { ...PLOT, ...(cfg.margin || {}) };
|
||||||
|
const width = Math.max(260, node.clientWidth || node.parentElement?.clientWidth || 640);
|
||||||
|
const w = width - margin.left - margin.right;
|
||||||
|
const h = height - margin.top - margin.bottom;
|
||||||
|
const svg = svgEl("svg", { viewBox: `0 0 ${width} ${height}`, width, height, role: "img" });
|
||||||
|
const g = svgEl("g", { transform: `translate(${margin.left},${margin.top})` });
|
||||||
|
svg.appendChild(g);
|
||||||
|
return { svg, g, w, h, width, height, margin, grid: cssVar("--border"), ink: cssVar("--text-secondary"),
|
||||||
|
muted: cssVar("--text-muted"), surface: cssVar("--surface-1") };
|
||||||
|
}
|
||||||
|
|
||||||
|
function yAxis(f, y, ticks, format) {
|
||||||
|
for (const t of ticks) {
|
||||||
|
const yy = y(t);
|
||||||
|
f.g.appendChild(svgEl("line", { x1: 0, x2: f.w, y1: yy, y2: yy, stroke: f.grid, "stroke-width": 1 }));
|
||||||
|
const lab = svgEl("text", { x: -8, y: yy + 3.5, "text-anchor": "end", fill: f.muted, "font-size": 10.5 });
|
||||||
|
lab.textContent = format(t);
|
||||||
|
f.g.appendChild(lab);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function xLabels(f, entries) { // entries: [{x, text}]
|
||||||
|
for (const e of entries) {
|
||||||
|
const t = svgEl("text", { x: e.x, y: f.h + 15, "text-anchor": "middle", fill: f.muted, "font-size": 10.5 });
|
||||||
|
t.textContent = e.text;
|
||||||
|
f.g.appendChild(t);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function legendFor(node, items) {
|
||||||
|
const box = document.createElement("div");
|
||||||
|
box.className = "legend";
|
||||||
|
for (const it of items) {
|
||||||
|
const s = document.createElement("span"); s.className = "item";
|
||||||
|
const k = document.createElement("span"); k.className = `key${it.rect ? " rect" : ""}`; k.style.background = it.color;
|
||||||
|
const n = document.createElement("span"); n.textContent = it.name;
|
||||||
|
s.append(k, n); box.appendChild(s);
|
||||||
|
}
|
||||||
|
node.appendChild(box);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Render `fn` into `node` now, and again on resize / theme change. */
|
||||||
|
function mount(node, fn) {
|
||||||
|
node.__render = fn;
|
||||||
|
fn();
|
||||||
|
if (!node.__observed) {
|
||||||
|
node.__observed = true;
|
||||||
|
const ro = new ResizeObserver(() => { if (node.isConnected) requestAnimationFrame(() => node.__render?.()); });
|
||||||
|
ro.observe(node);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function redrawAll(root = document) {
|
||||||
|
root.querySelectorAll(".chart").forEach((n) => n.__render?.());
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ line chart */
|
||||||
|
/** series: [{name, points: [[x, y], ...], color?}] ; xType 'time' | 'linear' */
|
||||||
|
export function lineChart(node, cfg) {
|
||||||
|
mount(node, () => {
|
||||||
|
node.replaceChildren();
|
||||||
|
const series = (cfg.series || []).filter((s) => s.points?.length);
|
||||||
|
if (!series.length) { node.appendChild(emptyBox("no data")); return; }
|
||||||
|
const f = frame(node, cfg);
|
||||||
|
const xs = series.flatMap((s) => s.points.map((p) => p[0]));
|
||||||
|
const ys = series.flatMap((s) => s.points.map((p) => p[1])).filter(Number.isFinite);
|
||||||
|
const [x0, x1] = extent(xs);
|
||||||
|
const [y0, y1] = padDomain(extent(ys), 0.08, cfg.includeZero);
|
||||||
|
const x = lin(x0, x1, 0, f.w), y = lin(y0, y1, f.h, 0);
|
||||||
|
const yFmt = cfg.yFormat || ((v) => v.toFixed(2));
|
||||||
|
yAxis(f, y, niceTicks(y0, y1, cfg.yTicks || 5), yFmt);
|
||||||
|
if (cfg.zeroLine && y0 < 0 && y1 > 0) {
|
||||||
|
f.g.appendChild(svgEl("line", { x1: 0, x2: f.w, y1: y(0), y2: y(0), stroke: cssVar("--border-strong"), "stroke-width": 1 }));
|
||||||
|
}
|
||||||
|
// x labels: ~6 evenly spaced actual data positions
|
||||||
|
const xFmt = cfg.xFormat || ((v) => String(Math.round(v)));
|
||||||
|
const uniq = [...new Set(xs)].sort((a, b) => a - b);
|
||||||
|
const n = Math.min(6, uniq.length);
|
||||||
|
const picks = Array.from({ length: n }, (_, i) => uniq[Math.round((i * (uniq.length - 1)) / Math.max(n - 1, 1))]);
|
||||||
|
xLabels(f, picks.map((v) => ({ x: x(v), text: xFmt(v) })));
|
||||||
|
|
||||||
|
series.forEach((s, i) => {
|
||||||
|
const color = s.color || seriesColor(i);
|
||||||
|
const d = s.points.filter((p) => Number.isFinite(p[1]))
|
||||||
|
.map((p, j) => `${j ? "L" : "M"}${x(p[0]).toFixed(2)},${y(p[1]).toFixed(2)}`).join(" ");
|
||||||
|
if (cfg.area) {
|
||||||
|
const pts = s.points.filter((p) => Number.isFinite(p[1]));
|
||||||
|
const base = y(Math.max(y0, Math.min(0, y1)));
|
||||||
|
const area = `${d} L${x(pts[pts.length - 1][0]).toFixed(2)},${base} L${x(pts[0][0]).toFixed(2)},${base} Z`;
|
||||||
|
f.g.appendChild(svgEl("path", { d: area, fill: color, "fill-opacity": 0.1, stroke: "none" }));
|
||||||
|
}
|
||||||
|
f.g.appendChild(svgEl("path", { d, fill: "none", stroke: color, "stroke-width": 2,
|
||||||
|
"stroke-linejoin": "round", "stroke-linecap": "round" }));
|
||||||
|
});
|
||||||
|
|
||||||
|
// crosshair + one tooltip listing every series
|
||||||
|
const hair = svgEl("line", { y1: 0, y2: f.h, stroke: cssVar("--border-strong"), "stroke-width": 1, opacity: 0 });
|
||||||
|
f.g.appendChild(hair);
|
||||||
|
const dots = series.map((s, i) => {
|
||||||
|
const c = svgEl("circle", { r: 4, fill: s.color || seriesColor(i), stroke: f.surface, "stroke-width": 2, opacity: 0 });
|
||||||
|
f.g.appendChild(c); return c;
|
||||||
|
});
|
||||||
|
const overlay = svgEl("rect", { x: 0, y: 0, width: f.w, height: f.h, fill: "transparent" });
|
||||||
|
f.g.appendChild(overlay);
|
||||||
|
const onMove = (ev) => {
|
||||||
|
const box = f.svg.getBoundingClientRect();
|
||||||
|
const px = ((ev.clientX - box.left) / box.width) * f.width - f.margin.left;
|
||||||
|
const target = x0 + (px / f.w) * (x1 - x0);
|
||||||
|
let best = uniq[0];
|
||||||
|
for (const v of uniq) if (Math.abs(v - target) < Math.abs(best - target)) best = v;
|
||||||
|
hair.setAttribute("x1", x(best)); hair.setAttribute("x2", x(best)); hair.setAttribute("opacity", 1);
|
||||||
|
const rows = [];
|
||||||
|
series.forEach((s, i) => {
|
||||||
|
const p = s.points.find((q) => q[0] === best);
|
||||||
|
dots[i].setAttribute("opacity", p && Number.isFinite(p[1]) ? 1 : 0);
|
||||||
|
if (p && Number.isFinite(p[1])) {
|
||||||
|
dots[i].setAttribute("cx", x(p[0])); dots[i].setAttribute("cy", y(p[1]));
|
||||||
|
rows.push({ color: s.color || seriesColor(i), name: s.name, value: yFmt(p[1]) });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
showTip(ev, xFmt(best), rows);
|
||||||
|
};
|
||||||
|
overlay.addEventListener("pointermove", onMove);
|
||||||
|
overlay.addEventListener("pointerleave", () => {
|
||||||
|
hair.setAttribute("opacity", 0); dots.forEach((d) => d.setAttribute("opacity", 0)); hideTip();
|
||||||
|
});
|
||||||
|
node.appendChild(f.svg);
|
||||||
|
if (series.length > 1) legendFor(node, series.map((s, i) => ({ name: s.name, color: s.color || seriesColor(i) })));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ bar chart */
|
||||||
|
/** items: [{label, values: [v, ...], meta?}] ; groups: [name, ...] (one entry = single series) */
|
||||||
|
export function barChart(node, cfg) {
|
||||||
|
mount(node, () => {
|
||||||
|
node.replaceChildren();
|
||||||
|
const items = cfg.items || [];
|
||||||
|
if (!items.length) { node.appendChild(emptyBox(cfg.emptyText || "nothing to show")); return; }
|
||||||
|
const groups = cfg.groups || [""];
|
||||||
|
const horizontal = cfg.horizontal !== false;
|
||||||
|
const fmt = cfg.valueFormat || ((v) => v.toFixed(2));
|
||||||
|
const all = items.flatMap((it) => it.values).filter(Number.isFinite);
|
||||||
|
const refs = (cfg.references || []).map((r) => r.value);
|
||||||
|
const dom = padDomain(extent([...all, ...refs]), 0.04, true);
|
||||||
|
const colorOf = (gi, v) => cfg.diverging ? (v < 0 ? cssVar("--div-neg") : cssVar("--div-pos"))
|
||||||
|
: (cfg.colors?.[gi] || seriesColor(gi));
|
||||||
|
|
||||||
|
if (horizontal) {
|
||||||
|
const rowH = Math.min(cfg.rowHeight || 26, 34);
|
||||||
|
const height = items.length * rowH + 34;
|
||||||
|
const f = frame(node, { ...cfg, height, margin: { ...PLOT, left: cfg.labelWidth || 92, bottom: 22, top: 6 } });
|
||||||
|
const x = lin(dom[0], dom[1], 0, f.w);
|
||||||
|
const band = f.h / items.length;
|
||||||
|
const thick = Math.min(24, Math.max(4, (band - 8) / groups.length - (groups.length > 1 ? 2 : 0)));
|
||||||
|
for (const t of niceTicks(dom[0], dom[1], 5)) {
|
||||||
|
f.g.appendChild(svgEl("line", { x1: x(t), x2: x(t), y1: 0, y2: f.h, stroke: f.grid, "stroke-width": 1 }));
|
||||||
|
const lab = svgEl("text", { x: x(t), y: f.h + 14, "text-anchor": "middle", fill: f.muted, "font-size": 10.5 });
|
||||||
|
lab.textContent = fmt(t); f.g.appendChild(lab);
|
||||||
|
}
|
||||||
|
items.forEach((it, i) => {
|
||||||
|
const y0 = i * band + (band - thick * groups.length - (groups.length - 1) * 2) / 2;
|
||||||
|
const lab = svgEl("text", { x: -8, y: i * band + band / 2 + 3.5, "text-anchor": "end",
|
||||||
|
fill: cssVar("--text-secondary"), "font-size": 11 });
|
||||||
|
lab.textContent = it.label; f.g.appendChild(lab);
|
||||||
|
it.values.forEach((v, gi) => {
|
||||||
|
if (!Number.isFinite(v)) return;
|
||||||
|
const yy = y0 + gi * (thick + 2);
|
||||||
|
const zero = x(Math.max(dom[0], Math.min(0, dom[1])));
|
||||||
|
const xv = x(v);
|
||||||
|
const r = svgEl("path", { d: roundedBar(Math.min(zero, xv), yy, Math.abs(xv - zero), thick, v >= 0 ? "right" : "left"),
|
||||||
|
fill: colorOf(gi, v) });
|
||||||
|
r.style.cursor = "default";
|
||||||
|
r.addEventListener("pointermove", (ev) => showTip(ev, it.label,
|
||||||
|
groups.map((g, k) => ({ color: colorOf(k, it.values[k]), name: g || (cfg.valueName || "value"),
|
||||||
|
value: Number.isFinite(it.values[k]) ? fmt(it.values[k]) : "-" }))));
|
||||||
|
r.addEventListener("pointerleave", hideTip);
|
||||||
|
f.g.appendChild(r);
|
||||||
|
if (cfg.valueLabels !== false && items.length <= 24 && groups.length === 1) {
|
||||||
|
const text = fmt(v);
|
||||||
|
const outside = v >= 0 ? xv + 5 : xv - 5;
|
||||||
|
const t = svgEl("text", { x: outside, y: yy + thick / 2 + 3.5, "text-anchor": v >= 0 ? "start" : "end",
|
||||||
|
fill: f.muted, "font-size": 10 });
|
||||||
|
t.textContent = text;
|
||||||
|
if (outside + (v >= 0 ? text.length * 5.6 : -text.length * 5.6) <= f.w + 4 &&
|
||||||
|
outside + (v >= 0 ? 0 : -text.length * 5.6) >= -4) f.g.appendChild(t);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
for (const r of cfg.references || []) drawRef(f, x(r.value), r, true);
|
||||||
|
node.appendChild(f.svg);
|
||||||
|
} else {
|
||||||
|
const f = frame(node, { ...cfg, height: cfg.height || 220, margin: { ...PLOT, bottom: cfg.bottom || 46 } });
|
||||||
|
const y = lin(dom[0], dom[1], f.h, 0);
|
||||||
|
const band = f.w / items.length;
|
||||||
|
const thick = Math.min(24, Math.max(3, (band - 6) / groups.length - (groups.length > 1 ? 2 : 0)));
|
||||||
|
yAxis(f, y, niceTicks(dom[0], dom[1], 5), fmt);
|
||||||
|
items.forEach((it, i) => {
|
||||||
|
const x0 = i * band + (band - thick * groups.length - (groups.length - 1) * 2) / 2;
|
||||||
|
it.values.forEach((v, gi) => {
|
||||||
|
if (!Number.isFinite(v)) return;
|
||||||
|
const zero = y(Math.max(dom[0], Math.min(0, dom[1])));
|
||||||
|
const yv = y(v);
|
||||||
|
const r = svgEl("path", { d: roundedBar(x0 + gi * (thick + 2), Math.min(zero, yv), thick, Math.abs(yv - zero),
|
||||||
|
v >= 0 ? "up" : "down", true), fill: colorOf(gi, v) });
|
||||||
|
r.addEventListener("pointermove", (ev) => showTip(ev, it.label,
|
||||||
|
groups.map((g, k) => ({ color: colorOf(k, it.values[k]), name: g || (cfg.valueName || "value"),
|
||||||
|
value: Number.isFinite(it.values[k]) ? fmt(it.values[k]) : "-" }))));
|
||||||
|
r.addEventListener("pointerleave", hideTip);
|
||||||
|
f.g.appendChild(r);
|
||||||
|
});
|
||||||
|
if (items.length <= 30) {
|
||||||
|
const t = svgEl("text", { x: i * band + band / 2, y: f.h + 14, "text-anchor": "end", fill: f.muted,
|
||||||
|
"font-size": 10, transform: `rotate(-35 ${i * band + band / 2} ${f.h + 14})` });
|
||||||
|
t.textContent = it.label; f.g.appendChild(t);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
node.appendChild(f.svg);
|
||||||
|
}
|
||||||
|
if (groups.length > 1) legendFor(node, groups.map((g, i) => ({ name: g, color: cfg.colors?.[i] || seriesColor(i), rect: true })));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function roundedBar(x, y, w, h, dir, vertical = false) {
|
||||||
|
const r = Math.min(4, vertical ? w / 2 : h / 2, vertical ? h : w);
|
||||||
|
if (w <= 0.5 || h <= 0.5) return `M${x},${y}h${Math.max(w, 0.5)}v${Math.max(h, 0.5)}h${-Math.max(w, 0.5)}Z`;
|
||||||
|
if (!vertical) {
|
||||||
|
return dir === "right"
|
||||||
|
? `M${x},${y} h${w - r} a${r},${r} 0 0 1 ${r},${r} v${h - 2 * r} a${r},${r} 0 0 1 ${-r},${r} h${-(w - r)} Z`
|
||||||
|
: `M${x + r},${y} h${w - r} v${h} h${-(w - r)} a${r},${r} 0 0 1 ${-r},${-r} v${-(h - 2 * r)} a${r},${r} 0 0 1 ${r},${-r} Z`;
|
||||||
|
}
|
||||||
|
return dir === "up"
|
||||||
|
? `M${x},${y + h} v${-(h - r)} a${r},${r} 0 0 1 ${r},${-r} h${w - 2 * r} a${r},${r} 0 0 1 ${r},${r} v${h - r} Z`
|
||||||
|
: `M${x},${y} h${w} v${h - r} a${r},${r} 0 0 1 ${-r},${r} h${-(w - 2 * r)} a${r},${r} 0 0 1 ${-r},${-r} Z`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function drawRef(f, pos, ref, vertical) {
|
||||||
|
const line = vertical
|
||||||
|
? svgEl("line", { x1: pos, x2: pos, y1: 0, y2: f.h, stroke: ref.color || cssVar("--text-muted"), "stroke-width": 1 })
|
||||||
|
: svgEl("line", { x1: 0, x2: f.w, y1: pos, y2: pos, stroke: ref.color || cssVar("--text-muted"), "stroke-width": 1 });
|
||||||
|
f.g.appendChild(line);
|
||||||
|
if (ref.label) {
|
||||||
|
const near = pos > f.w - Math.max(56, ref.label.length * 5.4); // keep the label inside the plot
|
||||||
|
const t = svgEl("text", { x: vertical ? (near ? pos - 4 : pos + 4) : f.w, y: vertical ? f.h - 4 : pos - 4,
|
||||||
|
"text-anchor": vertical ? (near ? "end" : "start") : "end",
|
||||||
|
fill: ref.color || cssVar("--text-muted"), "font-size": 10,
|
||||||
|
// a surface-coloured halo keeps it legible where it crosses a mark
|
||||||
|
stroke: f.surface, "stroke-width": 3, "paint-order": "stroke fill" });
|
||||||
|
t.textContent = ref.label; f.g.appendChild(t);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ histogram */
|
||||||
|
/** cfg: {edges, mass, markers: [{value, label, color}], tailFrom, valueFormat} */
|
||||||
|
export function histogram(node, cfg) {
|
||||||
|
mount(node, () => {
|
||||||
|
node.replaceChildren();
|
||||||
|
const { edges, mass } = cfg;
|
||||||
|
if (!edges?.length || !mass?.length) { node.appendChild(emptyBox("no distribution")); return; }
|
||||||
|
const f = frame(node, { ...cfg, height: cfg.height || 240, margin: { ...PLOT, left: 52, bottom: 28 } });
|
||||||
|
const x = lin(edges[0], edges[edges.length - 1], 0, f.w);
|
||||||
|
const ymax = Math.max(...mass);
|
||||||
|
const y = lin(0, ymax, f.h, 0);
|
||||||
|
const fmt = cfg.valueFormat || ((v) => v.toFixed(2));
|
||||||
|
const pct = (v) => `${(v * 100).toFixed(1)} %`;
|
||||||
|
for (const t of niceTicks(0, ymax, 4)) {
|
||||||
|
f.g.appendChild(svgEl("line", { x1: 0, x2: f.w, y1: y(t), y2: y(t), stroke: f.grid, "stroke-width": 1 }));
|
||||||
|
const lab = svgEl("text", { x: -8, y: y(t) + 3.5, "text-anchor": "end", fill: f.muted, "font-size": 10.5 });
|
||||||
|
lab.textContent = pct(t); f.g.appendChild(lab);
|
||||||
|
}
|
||||||
|
const main = cssVar("--series-1"), tail = cssVar("--series-2");
|
||||||
|
const tailFrom = cfg.tailFrom;
|
||||||
|
mass.forEach((m, i) => {
|
||||||
|
const x0 = x(edges[i]), x1 = x(edges[i + 1]);
|
||||||
|
const gap = Math.min(2, (x1 - x0) * 0.3);
|
||||||
|
const w = Math.max(x1 - x0 - gap, 0.6);
|
||||||
|
const isTail = tailFrom !== undefined && edges[i + 1] <= tailFrom;
|
||||||
|
const h = f.h - y(m);
|
||||||
|
const bar = svgEl("path", { d: roundedBar(x0 + gap / 2, y(m), w, h, "up", true), fill: isTail ? tail : main });
|
||||||
|
bar.addEventListener("pointermove", (ev) => showTip(ev, `${fmt(edges[i])} … ${fmt(edges[i + 1])}`,
|
||||||
|
[{ color: isTail ? tail : main, name: "probability", value: pct(m) }]));
|
||||||
|
bar.addEventListener("pointerleave", hideTip);
|
||||||
|
f.g.appendChild(bar);
|
||||||
|
});
|
||||||
|
for (const t of niceTicks(edges[0], edges[edges.length - 1], 6)) {
|
||||||
|
const lab = svgEl("text", { x: x(t), y: f.h + 15, "text-anchor": "middle", fill: f.muted, "font-size": 10.5 });
|
||||||
|
lab.textContent = fmt(t); f.g.appendChild(lab);
|
||||||
|
}
|
||||||
|
(cfg.markers || []).forEach((m, i) => {
|
||||||
|
const px = x(m.value);
|
||||||
|
f.g.appendChild(svgEl("line", { x1: px, x2: px, y1: 0, y2: f.h, stroke: m.color || cssVar("--text-primary"),
|
||||||
|
"stroke-width": 1 }));
|
||||||
|
const t = svgEl("text", { x: px + 3, y: 10 + i * 12, "text-anchor": px > f.w - 60 ? "end" : "start",
|
||||||
|
fill: m.color || cssVar("--text-secondary"), "font-size": 10 });
|
||||||
|
t.setAttribute("x", px > f.w - 60 ? px - 3 : px + 3);
|
||||||
|
t.textContent = m.label; f.g.appendChild(t);
|
||||||
|
});
|
||||||
|
node.appendChild(f.svg);
|
||||||
|
const items = [{ name: cfg.mainLabel || "outcomes", color: main, rect: true }];
|
||||||
|
if (tailFrom !== undefined) items.push({ name: cfg.tailLabel || "beyond VaR", color: tail, rect: true });
|
||||||
|
legendFor(node, items);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ heatmap */
|
||||||
|
/** cfg: {labels, matrix, diverging (default true), domain, format} */
|
||||||
|
export function heatmap(node, cfg) {
|
||||||
|
mount(node, () => {
|
||||||
|
node.replaceChildren();
|
||||||
|
const labels = cfg.labels || [], m = cfg.matrix || [];
|
||||||
|
if (!labels.length) { node.appendChild(emptyBox("no matrix")); return; }
|
||||||
|
const n = labels.length;
|
||||||
|
const width = Math.max(260, node.clientWidth || 600);
|
||||||
|
const pad = Math.min(96, Math.max(44, width * 0.14));
|
||||||
|
const cell = Math.max(6, Math.min(34, (width - pad - 66) / n));
|
||||||
|
const size = cell * n;
|
||||||
|
const f = { svg: svgEl("svg", { viewBox: `0 0 ${width} ${size + pad + 30}`, width, height: size + pad + 30 }),
|
||||||
|
muted: cssVar("--text-muted") };
|
||||||
|
const g = svgEl("g", { transform: `translate(${pad},${pad})` });
|
||||||
|
f.svg.appendChild(g);
|
||||||
|
const dom = cfg.domain || (cfg.diverging === false ? [0, Math.max(...m.flat())] : [-1, 1]);
|
||||||
|
const fmt = cfg.format || ((v) => v.toFixed(2));
|
||||||
|
const show = n <= 26;
|
||||||
|
for (let i = 0; i < n; i++) {
|
||||||
|
for (let j = 0; j < n; j++) {
|
||||||
|
const v = m[i]?.[j];
|
||||||
|
if (!Number.isFinite(v)) continue;
|
||||||
|
const c = colorFor(v, dom, cfg.diverging !== false);
|
||||||
|
const r = svgEl("rect", { x: j * cell, y: i * cell, width: Math.max(cell - 1, 1), height: Math.max(cell - 1, 1),
|
||||||
|
fill: c, rx: Math.min(2, cell / 6) });
|
||||||
|
r.addEventListener("pointermove", (ev) => showTip(ev, `${labels[i]} · ${labels[j]}`,
|
||||||
|
[{ color: c, name: cfg.valueName || "correlation", value: fmt(v) }]));
|
||||||
|
r.addEventListener("pointerleave", hideTip);
|
||||||
|
g.appendChild(r);
|
||||||
|
}
|
||||||
|
if (show) {
|
||||||
|
const ty = svgEl("text", { x: -6, y: i * cell + cell / 2 + 3, "text-anchor": "end", fill: f.muted, "font-size": 9.5 });
|
||||||
|
ty.textContent = labels[i]; g.appendChild(ty);
|
||||||
|
const tx = svgEl("text", { x: i * cell + cell / 2, y: -6, "text-anchor": "start", fill: f.muted, "font-size": 9.5,
|
||||||
|
transform: `rotate(-60 ${i * cell + cell / 2} -6)` });
|
||||||
|
tx.textContent = labels[i]; g.appendChild(tx);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// scale legend
|
||||||
|
const lw = Math.min(180, size), lx = pad, ly = pad + size + 18;
|
||||||
|
const grad = svgEl("linearGradient", { id: `hm-${Math.random().toString(36).slice(2)}` });
|
||||||
|
for (let k = 0; k <= 10; k++) {
|
||||||
|
grad.appendChild(svgEl("stop", { offset: `${k * 10}%`,
|
||||||
|
"stop-color": colorFor(dom[0] + (k / 10) * (dom[1] - dom[0]), dom, cfg.diverging !== false) }));
|
||||||
|
}
|
||||||
|
const defs = svgEl("defs"); defs.appendChild(grad); f.svg.appendChild(defs);
|
||||||
|
f.svg.appendChild(svgEl("rect", { x: lx, y: ly, width: lw, height: 8, rx: 2, fill: `url(#${grad.id})` }));
|
||||||
|
[dom[0], (dom[0] + dom[1]) / 2, dom[1]].forEach((v, k) => {
|
||||||
|
const t = svgEl("text", { x: lx + (k * lw) / 2, y: ly + 20, "text-anchor": k === 0 ? "start" : k === 2 ? "end" : "middle",
|
||||||
|
fill: f.muted, "font-size": 10 });
|
||||||
|
t.textContent = fmt(v); f.svg.appendChild(t);
|
||||||
|
});
|
||||||
|
node.appendChild(f.svg);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function hex2rgb(h) {
|
||||||
|
const s = h.replace("#", "").trim();
|
||||||
|
const v = s.length === 3 ? s.split("").map((c) => c + c).join("") : s;
|
||||||
|
return [parseInt(v.slice(0, 2), 16), parseInt(v.slice(2, 4), 16), parseInt(v.slice(4, 6), 16)];
|
||||||
|
}
|
||||||
|
const mix = (a, b, t) => `rgb(${a.map((v, i) => Math.round(v + (b[i] - v) * t)).join(",")})`;
|
||||||
|
|
||||||
|
export function colorFor(v, [lo, hi], diverging = true) {
|
||||||
|
if (diverging) {
|
||||||
|
const neg = hex2rgb(cssVar("--div-neg")), mid = hex2rgb(cssVar("--div-mid")), pos = hex2rgb(cssVar("--div-pos"));
|
||||||
|
const t = Math.max(-1, Math.min(1, v / Math.max(Math.abs(lo), Math.abs(hi) || 1)));
|
||||||
|
return t < 0 ? mix(mid, neg, -t) : mix(mid, pos, t);
|
||||||
|
}
|
||||||
|
const t = Math.max(0, Math.min(1, (v - lo) / ((hi - lo) || 1)));
|
||||||
|
const stops = ["--seq-100", "--seq-250", "--seq-400", "--seq-550", "--seq-700"].map((s) => hex2rgb(cssVar(s)));
|
||||||
|
const k = Math.min(stops.length - 2, Math.floor(t * (stops.length - 1)));
|
||||||
|
return mix(stops[k], stops[k + 1], t * (stops.length - 1) - k);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ scatter */
|
||||||
|
/** cfg: {points: [{x, y, label, group}], groups, xFormat, yFormat, xLabel, yLabel} */
|
||||||
|
export function scatterChart(node, cfg) {
|
||||||
|
mount(node, () => {
|
||||||
|
node.replaceChildren();
|
||||||
|
const pts = (cfg.points || []).filter((p) => Number.isFinite(p.x) && Number.isFinite(p.y));
|
||||||
|
if (!pts.length) { node.appendChild(emptyBox("no points")); return; }
|
||||||
|
const f = frame(node, { ...cfg, height: cfg.height || 260, margin: { ...PLOT, bottom: 40, left: 62 } });
|
||||||
|
const [x0, x1] = padDomain(extent(pts.map((p) => p.x)), 0.08);
|
||||||
|
const [y0, y1] = padDomain(extent(pts.map((p) => p.y)), 0.08);
|
||||||
|
const x = lin(x0, x1, 0, f.w), y = lin(y0, y1, f.h, 0);
|
||||||
|
const xFmt = cfg.xFormat || ((v) => v.toFixed(2)), yFmt = cfg.yFormat || ((v) => v.toFixed(2));
|
||||||
|
yAxis(f, y, niceTicks(y0, y1, 5), yFmt);
|
||||||
|
for (const t of niceTicks(x0, x1, 5)) {
|
||||||
|
f.g.appendChild(svgEl("line", { x1: x(t), x2: x(t), y1: 0, y2: f.h, stroke: f.grid, "stroke-width": 1 }));
|
||||||
|
const lab = svgEl("text", { x: x(t), y: f.h + 15, "text-anchor": "middle", fill: f.muted, "font-size": 10.5 });
|
||||||
|
lab.textContent = xFmt(t); f.g.appendChild(lab);
|
||||||
|
}
|
||||||
|
if (cfg.xLabel) {
|
||||||
|
const t = svgEl("text", { x: f.w / 2, y: f.h + 33, "text-anchor": "middle", fill: f.muted, "font-size": 10.5 });
|
||||||
|
t.textContent = cfg.xLabel; f.g.appendChild(t);
|
||||||
|
}
|
||||||
|
if (cfg.yLabel) {
|
||||||
|
const t = svgEl("text", { x: -f.margin.left + 12, y: f.h / 2, "text-anchor": "middle", fill: f.muted, "font-size": 10.5,
|
||||||
|
transform: `rotate(-90 ${-f.margin.left + 12} ${f.h / 2})` });
|
||||||
|
t.textContent = cfg.yLabel; f.g.appendChild(t);
|
||||||
|
}
|
||||||
|
const groups = cfg.groups || [...new Set(pts.map((p) => p.group ?? ""))];
|
||||||
|
pts.forEach((p) => {
|
||||||
|
const gi = Math.max(0, groups.indexOf(p.group ?? ""));
|
||||||
|
const c = p.color || seriesColor(gi);
|
||||||
|
const dot = svgEl("circle", { cx: x(p.x), cy: y(p.y), r: 5, fill: c, stroke: f.surface, "stroke-width": 2 });
|
||||||
|
const hit = svgEl("circle", { cx: x(p.x), cy: y(p.y), r: 12, fill: "transparent" });
|
||||||
|
hit.addEventListener("pointermove", (ev) => showTip(ev, p.label, [
|
||||||
|
{ color: c, name: cfg.xLabel || "x", value: xFmt(p.x) }, { color: c, name: cfg.yLabel || "y", value: yFmt(p.y) },
|
||||||
|
]));
|
||||||
|
hit.addEventListener("pointerleave", hideTip);
|
||||||
|
f.g.append(dot, hit);
|
||||||
|
});
|
||||||
|
if (cfg.labelPoints !== false && pts.length <= 14) {
|
||||||
|
pts.forEach((p) => {
|
||||||
|
const t = svgEl("text", { x: x(p.x) + 8, y: y(p.y) - 7, fill: f.muted, "font-size": 9.5 });
|
||||||
|
t.textContent = p.label; f.g.appendChild(t);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
node.appendChild(f.svg);
|
||||||
|
if (groups.length > 1) legendFor(node, groups.map((g, i) => ({ name: g, color: seriesColor(i) })));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function emptyBox(text) {
|
||||||
|
const d = document.createElement("div");
|
||||||
|
d.className = "empty"; d.textContent = text;
|
||||||
|
return d;
|
||||||
|
}
|
||||||
393
quantcore/api/ui/js/core.js
Normal file
393
quantcore/api/ui/js/core.js
Normal file
@ -0,0 +1,393 @@
|
|||||||
|
/* Shared state, API client, formatting and DOM helpers for the test bench. */
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ API */
|
||||||
|
const BASE = location.pathname.replace(/\/ui\/?.*$/, "");
|
||||||
|
|
||||||
|
async function request(path, { method = "GET", body, signal } = {}) {
|
||||||
|
const res = await fetch(BASE + path, {
|
||||||
|
method,
|
||||||
|
headers: body ? { "Content-Type": "application/json" } : undefined,
|
||||||
|
body: body ? JSON.stringify(body) : undefined,
|
||||||
|
signal,
|
||||||
|
});
|
||||||
|
const text = await res.text();
|
||||||
|
let data = null;
|
||||||
|
try { data = text ? JSON.parse(text) : null; } catch { data = text; }
|
||||||
|
if (!res.ok) {
|
||||||
|
const detail = data && typeof data === "object" ? data.detail ?? JSON.stringify(data) : String(data || res.statusText);
|
||||||
|
throw new Error(typeof detail === "string" ? detail : JSON.stringify(detail));
|
||||||
|
}
|
||||||
|
return data;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const api = {
|
||||||
|
get: (path, params) => request(path + (params ? "?" + new URLSearchParams(
|
||||||
|
Object.entries(params).filter(([, v]) => v !== null && v !== undefined && v !== "")).toString() : "")),
|
||||||
|
post: (path, body, opts = {}) => request(path, { method: "POST", body, ...opts }),
|
||||||
|
};
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ state */
|
||||||
|
const KEY = "quantcore-bench-v2";
|
||||||
|
|
||||||
|
export const DEFAULT_STATE = {
|
||||||
|
ui: { tab: "stocks", theme: "auto", migrations: [] },
|
||||||
|
universe: { region: "", sector: "", country: "", search: "", top: 30 },
|
||||||
|
selection: [],
|
||||||
|
portfolio: { mode: "new", cash: 1000000, holdings: {} },
|
||||||
|
model: {
|
||||||
|
freq: "daily", horizon: "week", scenario_model: "gbm", n_scenarios: 4096, seed: 42,
|
||||||
|
sampling: "sobol", moment_match: true, importance_sampling: "auto", drift_convention: "arithmetic",
|
||||||
|
innovation_history: 2500,
|
||||||
|
covariance: { method: "ewm", lam: 0.99, window: 200, shrinkage: "auto" },
|
||||||
|
expected_returns: "black_litterman", bl_tau: 0.05, bl_years: 10, bl_omega_method: "idzorek",
|
||||||
|
},
|
||||||
|
views: [],
|
||||||
|
constraints: {
|
||||||
|
max_weight: 0.1, max_sector_weight: 0.25, sector_overrides: {}, max_positions: null,
|
||||||
|
min_position_value: 0, max_turnover: null, max_transaction_cost: null, min_invested: 0, max_invested: 1,
|
||||||
|
},
|
||||||
|
costs: { proportional_buy: 0.001, proportional_sell: 0.001, fixed_per_trade: 0 },
|
||||||
|
opt: {
|
||||||
|
objective: "max_return", confidence: 0.95, cvar_mode: "fraction", cvar_limit_fraction: 0.05,
|
||||||
|
cvar_limit: null, risk_aversion: 1, target_return: null, solver: "auto", time_limit: 300,
|
||||||
|
row_generation: "auto", auto_scenarios: true, risk_drift: "model",
|
||||||
|
},
|
||||||
|
data: { start: "", end: "", freq: "daily" },
|
||||||
|
};
|
||||||
|
|
||||||
|
function deepMerge(base, add) {
|
||||||
|
if (Array.isArray(base) || add === null || typeof add !== "object" || typeof base !== "object") return add ?? base;
|
||||||
|
const out = { ...base };
|
||||||
|
for (const [k, v] of Object.entries(add)) out[k] = k in base ? deepMerge(base[k], v) : v;
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
function load() {
|
||||||
|
try {
|
||||||
|
const raw = localStorage.getItem(KEY);
|
||||||
|
return raw ? migrate(deepMerge(structuredClone(DEFAULT_STATE), JSON.parse(raw))) : structuredClone(DEFAULT_STATE);
|
||||||
|
} catch { return structuredClone(DEFAULT_STATE); }
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Fixes for settings stored by an earlier version (each applied once). */
|
||||||
|
function migrate(s) {
|
||||||
|
s.ui.migrations = s.ui.migrations || [];
|
||||||
|
if (!s.ui.migrations.includes("time-limit")) {
|
||||||
|
// a stored `null` meant "no limit", and without one HiGHS can run for hours on a large problem
|
||||||
|
if (s.opt.time_limit === null) s.opt.time_limit = DEFAULT_STATE.opt.time_limit;
|
||||||
|
s.ui.migrations.push("time-limit");
|
||||||
|
}
|
||||||
|
return s;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const state = load();
|
||||||
|
|
||||||
|
const listeners = new Set();
|
||||||
|
export function onChange(fn) { listeners.add(fn); return () => listeners.delete(fn); }
|
||||||
|
|
||||||
|
/** Persist and notify. `topic` says what changed ('selection', 'model', ...); `source` names the
|
||||||
|
tab that changed it, so that tab can update itself without being re-rendered from outside. */
|
||||||
|
export function changed(topic = "state", source = null) {
|
||||||
|
try { localStorage.setItem(KEY, JSON.stringify(state)); } catch { /* private mode: keep going */ }
|
||||||
|
for (const fn of listeners) fn(topic, source);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function resetState() {
|
||||||
|
try { localStorage.removeItem(KEY); } catch { /* ignore */ }
|
||||||
|
location.reload();
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ requests built from the state */
|
||||||
|
export function covarianceSpec() {
|
||||||
|
const c = state.model.covariance;
|
||||||
|
return { method: c.method, lam: Number(c.lam), window: Number(c.window),
|
||||||
|
shrinkage: c.shrinkage === "auto" ? "auto" : Number(c.shrinkage) };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function viewPayload() {
|
||||||
|
return state.views.map((v) => (v.kind === "sentiment"
|
||||||
|
? { symbol: v.symbol, sentiment: Number(v.sentiment), confidence: Number(v.confidence),
|
||||||
|
strength: Number(v.strength), label: v.label || "", source: v.source || "" }
|
||||||
|
: { assets: v.assets, q: Number(v.q), confidence: Number(v.confidence), label: v.label || "", source: v.source || "" }));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function constraintPayload() {
|
||||||
|
const c = state.constraints;
|
||||||
|
const overrides = Object.entries(c.sector_overrides || {}).filter(([, v]) => v !== "" && v !== null);
|
||||||
|
return {
|
||||||
|
max_weight: Number(c.max_weight),
|
||||||
|
max_sector_weight: overrides.length
|
||||||
|
? { "*": Number(c.max_sector_weight), ...Object.fromEntries(overrides.map(([k, v]) => [k, Number(v)])) }
|
||||||
|
: Number(c.max_sector_weight),
|
||||||
|
max_positions: c.max_positions ? Number(c.max_positions) : null,
|
||||||
|
min_position_value: Number(c.min_position_value) || 0,
|
||||||
|
max_turnover: c.max_turnover === null || c.max_turnover === "" ? null : Number(c.max_turnover),
|
||||||
|
max_transaction_cost: c.max_transaction_cost === null || c.max_transaction_cost === "" ? null : Number(c.max_transaction_cost),
|
||||||
|
min_invested: Number(c.min_invested) || 0,
|
||||||
|
max_invested: Number(c.max_invested) || 1,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function scenarioPayload(extra = {}) {
|
||||||
|
const m = state.model;
|
||||||
|
return {
|
||||||
|
symbols: state.selection, model: m.scenario_model === "gaussian_exact" ? "gbm" : m.scenario_model,
|
||||||
|
n_scenarios: Number(m.n_scenarios), horizon: m.horizon, freq: m.freq, covariance: covarianceSpec(),
|
||||||
|
seed: Number(m.seed), sampling: m.sampling, moment_match: !!m.moment_match,
|
||||||
|
drift_convention: m.drift_convention, expected_returns: m.expected_returns,
|
||||||
|
innovation_history: m.innovation_history ? Number(m.innovation_history) : null,
|
||||||
|
importance_sampling: m.importance_sampling, views: viewPayload(),
|
||||||
|
bl_tau: Number(m.bl_tau), bl_years: Number(m.bl_years),
|
||||||
|
bl_omega_method: m.bl_omega_method || "idzorek", ...extra,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function optimizePayload(extra = {}) {
|
||||||
|
const m = state.model, o = state.opt, p = state.portfolio;
|
||||||
|
const body = {
|
||||||
|
mode: p.mode, cash: Number(p.cash), holdings: p.mode === "rebalance" ? intHoldings(p.holdings) : {},
|
||||||
|
symbols: state.selection.length ? state.selection : null, max_assets: Math.max(state.selection.length || 30, 2),
|
||||||
|
horizon: m.horizon, freq: m.freq, confidence: Number(o.confidence),
|
||||||
|
objective: o.objective, risk_aversion: Number(o.risk_aversion),
|
||||||
|
scenario_model: m.scenario_model, seed: Number(m.seed), sampling: m.sampling,
|
||||||
|
moment_match: !!m.moment_match, importance_sampling: m.importance_sampling,
|
||||||
|
drift_convention: m.drift_convention, row_generation: o.row_generation, risk_drift: o.risk_drift,
|
||||||
|
covariance: covarianceSpec(), expected_returns: m.expected_returns, views: viewPayload(),
|
||||||
|
bl_tau: Number(m.bl_tau), bl_years: Number(m.bl_years), bl_omega_method: m.bl_omega_method || "idzorek",
|
||||||
|
constraints: constraintPayload(), costs: {
|
||||||
|
proportional_buy: Number(state.costs.proportional_buy), proportional_sell: Number(state.costs.proportional_sell),
|
||||||
|
fixed_per_trade: Number(state.costs.fixed_per_trade),
|
||||||
|
},
|
||||||
|
solver: o.solver, time_limit: o.time_limit ? Number(o.time_limit) : null,
|
||||||
|
innovation_history: m.innovation_history ? Number(m.innovation_history) : null,
|
||||||
|
};
|
||||||
|
if (!o.auto_scenarios && m.scenario_model !== "gaussian_exact") body.n_scenarios = Number(m.n_scenarios);
|
||||||
|
if (o.objective === "max_return") {
|
||||||
|
if (o.cvar_mode === "money" && o.cvar_limit) body.cvar_limit = Number(o.cvar_limit);
|
||||||
|
else body.cvar_limit_fraction = Number(o.cvar_limit_fraction);
|
||||||
|
} else if (o.objective === "min_cvar") {
|
||||||
|
body.target_return = o.target_return === null || o.target_return === "" ? null : Number(o.target_return);
|
||||||
|
}
|
||||||
|
return { ...body, ...extra }; // `extra` wins: a sweep overrides exactly the field it varies
|
||||||
|
}
|
||||||
|
|
||||||
|
export function intHoldings(h) {
|
||||||
|
return Object.fromEntries(Object.entries(h || {}).map(([k, v]) => [k, Math.round(Number(v) || 0)]).filter(([, v]) => v > 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ formatting */
|
||||||
|
const nf = (min, max) => new Intl.NumberFormat("de-CH", { minimumFractionDigits: min, maximumFractionDigits: max });
|
||||||
|
|
||||||
|
export const fmt = {
|
||||||
|
money: (v, d = 0) => (Number.isFinite(v) ? nf(d, d).format(v) : "–"),
|
||||||
|
money2: (v) => (Number.isFinite(v) ? nf(2, 2).format(v) : "–"),
|
||||||
|
signed: (v, d = 0) => (Number.isFinite(v) ? (v > 0 ? "+" : "") + nf(d, d).format(v) : "–"),
|
||||||
|
compact: (v) => {
|
||||||
|
if (!Number.isFinite(v)) return "–";
|
||||||
|
const a = Math.abs(v);
|
||||||
|
if (a >= 1e12) return `${nf(2, 2).format(v / 1e12)} T`;
|
||||||
|
if (a >= 1e9) return `${nf(1, 1).format(v / 1e9)} bn`;
|
||||||
|
if (a >= 1e6) return `${nf(1, 1).format(v / 1e6)} m`;
|
||||||
|
if (a >= 1e3) return `${nf(1, 1).format(v / 1e3)} k`;
|
||||||
|
return nf(0, 1).format(v);
|
||||||
|
},
|
||||||
|
pct: (v, d = 1) => (Number.isFinite(v) ? `${nf(d, d).format(v * 100)} %` : "–"),
|
||||||
|
pctSigned: (v, d = 1) => (Number.isFinite(v) ? `${v > 0 ? "+" : ""}${nf(d, d).format(v * 100)} %` : "–"),
|
||||||
|
bp: (v) => (Number.isFinite(v) ? `${nf(0, 1).format(v * 10000)} bp` : "–"),
|
||||||
|
num: (v, d = 2) => (Number.isFinite(v) ? nf(d, d).format(v) : "–"),
|
||||||
|
int: (v) => (Number.isFinite(v) ? nf(0, 0).format(Math.round(v)) : "–"),
|
||||||
|
date: (v) => (typeof v === "string" ? v.slice(0, 10) : new Date(v).toISOString().slice(0, 10)),
|
||||||
|
secs: (v) => (Number.isFinite(v) ? `${nf(1, 1).format(v)} s` : "–"),
|
||||||
|
};
|
||||||
|
|
||||||
|
export const parseDate = (s) => Date.parse(String(s).slice(0, 10) + "T00:00:00Z");
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ DOM helpers */
|
||||||
|
export function el(tag, props = {}, ...children) {
|
||||||
|
const e = document.createElement(tag);
|
||||||
|
for (const [k, v] of Object.entries(props || {})) {
|
||||||
|
if (v === null || v === undefined) continue;
|
||||||
|
if (k === "class") e.className = v;
|
||||||
|
else if (k === "text") e.textContent = v;
|
||||||
|
else if (k === "html") e.innerHTML = v; // only for constants in this file
|
||||||
|
else if (k === "style") Object.assign(e.style, v);
|
||||||
|
else if (k.startsWith("on") && typeof v === "function") e.addEventListener(k.slice(2), v);
|
||||||
|
else if (k === "value") e.value = v;
|
||||||
|
else if (typeof v === "boolean") { if (v) e.setAttribute(k, ""); }
|
||||||
|
else e.setAttribute(k, String(v));
|
||||||
|
}
|
||||||
|
for (const c of children.flat()) {
|
||||||
|
if (c === null || c === undefined || c === false) continue;
|
||||||
|
e.append(c instanceof Node ? c : document.createTextNode(String(c)));
|
||||||
|
}
|
||||||
|
return e;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** `replaceChildren` that drops null/false entries (so `cond ? node : null` is safe). */
|
||||||
|
export function setChildren(node, ...kids) {
|
||||||
|
node.replaceChildren(...kids.flat().filter((k) => k !== null && k !== undefined && k !== false));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function card(title, sub, ...content) {
|
||||||
|
const head = el("header", {}, title ? el("h2", { text: title }) : null, sub ? el("span", { class: "sub", text: sub }) : null);
|
||||||
|
return el("section", { class: "card" }, title || sub ? head : null, ...content);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function field(label, control, hint) {
|
||||||
|
return el("div", { class: "field" }, el("label", { text: label }), control,
|
||||||
|
hint ? el("span", { class: "hint", text: hint }) : null);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function input(type, value, onInput, attrs = {}) {
|
||||||
|
const e = el("input", { type, value: value ?? "", ...attrs });
|
||||||
|
e.addEventListener("change", () => onInput(type === "number" ? (e.value === "" ? null : Number(e.value)) : e.value));
|
||||||
|
return e;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function select(options, value, onInput, attrs = {}) {
|
||||||
|
const e = el("select", attrs);
|
||||||
|
for (const o of options) {
|
||||||
|
const [v, label] = Array.isArray(o) ? o : [o, o];
|
||||||
|
e.appendChild(el("option", { value: v, text: label, selected: String(v) === String(value) }));
|
||||||
|
}
|
||||||
|
e.value = value;
|
||||||
|
e.addEventListener("change", () => onInput(e.value));
|
||||||
|
return e;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function checkbox(label, value, onInput) {
|
||||||
|
const box = el("input", { type: "checkbox" });
|
||||||
|
box.checked = !!value;
|
||||||
|
box.addEventListener("change", () => onInput(box.checked));
|
||||||
|
return el("label", { class: "check" }, box, el("span", { text: label }));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function segmented(options, value, onInput) {
|
||||||
|
const wrap = el("div", { class: "seg" });
|
||||||
|
for (const o of options) {
|
||||||
|
const [v, label] = Array.isArray(o) ? o : [o, o];
|
||||||
|
const b = el("button", { type: "button", text: label, "aria-pressed": String(v) === String(value) });
|
||||||
|
b.addEventListener("click", () => onInput(v));
|
||||||
|
wrap.appendChild(b);
|
||||||
|
}
|
||||||
|
return wrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function button(label, onClick, cls = "btn", title = "") {
|
||||||
|
return el("button", { class: cls, type: "button", text: label, onclick: onClick, title });
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Sortable table. columns: [{key, label, num, fmt, cls, title}] */
|
||||||
|
export function dataTable(columns, rows, opts = {}) {
|
||||||
|
const wrap = el("div", { class: "table-wrap" });
|
||||||
|
const table = el("table", { class: "data" });
|
||||||
|
const thead = el("thead"), tbody = el("tbody");
|
||||||
|
let sortKey = opts.sortKey ?? null, sortDir = opts.sortDir ?? -1;
|
||||||
|
|
||||||
|
const headRow = el("tr");
|
||||||
|
for (const c of columns) {
|
||||||
|
const th = el("th", { class: `${c.num ? "num" : ""}${opts.sortable === false ? "" : " sortable"}`, title: c.title || "" });
|
||||||
|
th.append(document.createTextNode(c.label));
|
||||||
|
if (opts.sortable !== false) {
|
||||||
|
th.appendChild(el("span", { class: "arrow", text: sortKey === c.key ? (sortDir < 0 ? " ▼" : " ▲") : "" }));
|
||||||
|
th.addEventListener("click", () => {
|
||||||
|
if (sortKey === c.key) sortDir = -sortDir; else { sortKey = c.key; sortDir = c.num ? -1 : 1; }
|
||||||
|
render();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
headRow.appendChild(th);
|
||||||
|
}
|
||||||
|
thead.appendChild(headRow);
|
||||||
|
|
||||||
|
function render() {
|
||||||
|
const data = [...rows];
|
||||||
|
if (sortKey) {
|
||||||
|
data.sort((a, b) => {
|
||||||
|
const x = a[sortKey], y = b[sortKey];
|
||||||
|
if (x === y) return 0;
|
||||||
|
if (x === null || x === undefined) return 1;
|
||||||
|
if (y === null || y === undefined) return -1;
|
||||||
|
return (typeof x === "number" && typeof y === "number" ? x - y : String(x).localeCompare(String(y))) * sortDir;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
tbody.replaceChildren();
|
||||||
|
for (const r of data.slice(0, opts.limit || 5000)) {
|
||||||
|
const tr = el("tr", opts.rowClass ? { class: opts.rowClass(r) || "" } : {});
|
||||||
|
if (opts.onRowClick) { tr.style.cursor = "pointer"; tr.addEventListener("click", () => opts.onRowClick(r)); }
|
||||||
|
for (const c of columns) {
|
||||||
|
const td = el("td", { class: [c.num ? "num" : "", c.cls ? (typeof c.cls === "function" ? c.cls(r) : c.cls) : ""].join(" ").trim() });
|
||||||
|
const v = c.render ? c.render(r) : (c.fmt ? c.fmt(r[c.key], r) : r[c.key]);
|
||||||
|
if (v instanceof Node) td.appendChild(v);
|
||||||
|
else td.textContent = v === null || v === undefined ? "–" : String(v);
|
||||||
|
tr.appendChild(td);
|
||||||
|
}
|
||||||
|
tbody.appendChild(tr);
|
||||||
|
}
|
||||||
|
headRow.querySelectorAll(".arrow").forEach((a, i) => {
|
||||||
|
a.textContent = sortKey === columns[i].key ? (sortDir < 0 ? " ▼" : " ▲") : "";
|
||||||
|
});
|
||||||
|
}
|
||||||
|
render();
|
||||||
|
table.append(thead, tbody);
|
||||||
|
wrap.appendChild(table);
|
||||||
|
return wrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function statTiles(items) {
|
||||||
|
return el("div", { class: "stats" }, ...items.map((s) => el("div", { class: "stat" },
|
||||||
|
el("div", { class: "label", text: s.label }),
|
||||||
|
el("div", { class: "value", text: s.value, title: s.title || "" }),
|
||||||
|
s.delta ? el("div", { class: `delta ${s.deltaClass || "muted"}`, text: s.delta }) : null)));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function kv(pairs) {
|
||||||
|
const dl = el("dl", { class: "kv" });
|
||||||
|
for (const [k, v] of pairs) {
|
||||||
|
if (v === null || v === undefined) continue;
|
||||||
|
dl.append(el("dt", { text: k }), el("dd", { text: String(v) }));
|
||||||
|
}
|
||||||
|
return dl;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The panel currently in the document for `id` -- a re-render elsewhere may have replaced the
|
||||||
|
node a pending request captured, and writing a result into a detached node loses it. */
|
||||||
|
export function live(id, fallback) {
|
||||||
|
return document.getElementById(id) || fallback;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function chartBox(height) {
|
||||||
|
return el("div", { class: "chart", style: height ? { minHeight: `${height}px` } : {} });
|
||||||
|
}
|
||||||
|
|
||||||
|
export function warnBox(messages) {
|
||||||
|
if (!messages?.length) return null;
|
||||||
|
return el("div", {}, ...messages.map((m) => el("div", { class: "warn-box", text: m })));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function errorBox(err) {
|
||||||
|
return el("div", { class: "error-box", text: String(err?.message || err) });
|
||||||
|
}
|
||||||
|
|
||||||
|
export function empty(text) { return el("div", { class: "empty", text }); }
|
||||||
|
|
||||||
|
/** Run an async task, showing a spinner in `btn` and errors in `target`. */
|
||||||
|
export async function withBusy(btn, target, fn) {
|
||||||
|
const label = btn?.textContent;
|
||||||
|
if (btn) { btn.disabled = true; btn.replaceChildren(el("span", { class: "spinner" }), document.createTextNode(" " + label)); }
|
||||||
|
target?.classList.add("busy");
|
||||||
|
try {
|
||||||
|
return await fn();
|
||||||
|
} catch (err) {
|
||||||
|
target?.replaceChildren(errorBox(err));
|
||||||
|
throw err;
|
||||||
|
} finally {
|
||||||
|
if (btn) { btn.disabled = false; btn.textContent = label; }
|
||||||
|
target?.classList.remove("busy");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export const HORIZONS = [["day", "1 day"], ["week", "1 week"], ["month", "1 month"], ["quarter", "1 quarter"], ["year", "1 year"]];
|
||||||
|
export const SECTOR_ORDER = [
|
||||||
|
"Information Technology", "Financials", "Health Care", "Consumer Discretionary", "Industrials",
|
||||||
|
"Communication Services", "Consumer Staples", "Energy", "Materials", "Utilities", "Real Estate", "Unknown",
|
||||||
|
];
|
||||||
85
quantcore/api/ui/js/main.js
Normal file
85
quantcore/api/ui/js/main.js
Normal file
@ -0,0 +1,85 @@
|
|||||||
|
/* Entry point: the tab frame, the theme and the wiring between the sidebar and the tabs. */
|
||||||
|
|
||||||
|
import { changed, el, onChange, state } from "./core.js";
|
||||||
|
import { redrawAll } from "./charts.js";
|
||||||
|
import { buildSidebar, refreshHealth } from "./sidebar.js";
|
||||||
|
import { universe } from "./universe.js";
|
||||||
|
import * as stocks from "./tabs/stocks.js";
|
||||||
|
import * as risk from "./tabs/risk.js";
|
||||||
|
import * as views from "./tabs/views.js";
|
||||||
|
import * as constraints from "./tabs/constraints.js";
|
||||||
|
import * as optimize from "./tabs/optimize.js";
|
||||||
|
|
||||||
|
const TABS = [
|
||||||
|
{ id: "stocks", label: "Stocks & data", mod: stocks, deps: ["selection", "universe", "data", "model"] },
|
||||||
|
{ id: "risk", label: "Risk & scenarios", mod: risk, deps: ["selection", "model", "portfolio", "views"] },
|
||||||
|
{ id: "views", label: "Expected returns", mod: views, deps: ["selection", "model", "views"] },
|
||||||
|
{ id: "constraints", label: "Constraints", mod: constraints, deps: ["selection", "constraints", "costs", "portfolio"] },
|
||||||
|
{ id: "optimize", label: "Optimisation", mod: optimize, deps: ["selection", "model", "portfolio", "opt", "constraints", "costs", "views"] },
|
||||||
|
];
|
||||||
|
|
||||||
|
const tabBar = document.getElementById("tabs");
|
||||||
|
const main = document.getElementById("main");
|
||||||
|
|
||||||
|
function activate(id, force = false) {
|
||||||
|
if (!force && state.ui.tab === id && main.dataset.tab === id) return;
|
||||||
|
state.ui.tab = id;
|
||||||
|
changed("ui");
|
||||||
|
main.dataset.tab = id;
|
||||||
|
for (const b of tabBar.children) b.setAttribute("aria-selected", String(b.dataset.tab === id));
|
||||||
|
const tab = TABS.find((t) => t.id === id) || TABS[0];
|
||||||
|
main.replaceChildren(el("div", { class: "tabpanel" }));
|
||||||
|
const panel = main.firstChild;
|
||||||
|
Promise.resolve(tab.mod.render(panel)).catch((err) => {
|
||||||
|
panel.replaceChildren(el("div", { class: "error-box", text: String(err?.message || err) }));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const t of TABS) {
|
||||||
|
const b = el("button", { type: "button", role: "tab", text: t.label, "data-tab": t.id, "aria-selected": "false" });
|
||||||
|
b.addEventListener("click", () => activate(t.id, true));
|
||||||
|
tabBar.appendChild(b);
|
||||||
|
}
|
||||||
|
|
||||||
|
// a tab can send the reader to another tab ("edit these constraints")
|
||||||
|
document.addEventListener("quantcore:tab", (ev) => {
|
||||||
|
if (TABS.some((t) => t.id === ev.detail)) activate(ev.detail, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ theme */
|
||||||
|
const themeBtn = document.getElementById("theme");
|
||||||
|
function applyTheme() {
|
||||||
|
const t = state.ui.theme;
|
||||||
|
if (t === "auto") document.documentElement.removeAttribute("data-theme");
|
||||||
|
else document.documentElement.setAttribute("data-theme", t);
|
||||||
|
themeBtn.textContent = t === "auto" ? "Theme: system" : t === "dark" ? "Theme: dark" : "Theme: light";
|
||||||
|
requestAnimationFrame(() => redrawAll());
|
||||||
|
}
|
||||||
|
themeBtn.addEventListener("click", () => {
|
||||||
|
state.ui.theme = { auto: "light", light: "dark", dark: "auto" }[state.ui.theme] || "auto";
|
||||||
|
changed("ui");
|
||||||
|
applyTheme();
|
||||||
|
});
|
||||||
|
applyTheme();
|
||||||
|
matchMedia("(prefers-color-scheme: dark)").addEventListener("change", () => { if (state.ui.theme === "auto") applyTheme(); });
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ start */
|
||||||
|
buildSidebar(document.getElementById("sidebar"));
|
||||||
|
refreshHealth(document.getElementById("status"));
|
||||||
|
activate(state.ui.tab || "stocks", true);
|
||||||
|
|
||||||
|
// Every tab reads names, sectors and prices synchronously through `info()`, but only the Stocks tab
|
||||||
|
// used to fetch the universe -- so a reload that restored any other tab (the tab is part of the
|
||||||
|
// stored state) drew blank names and a price of 0, and the holdings editor showed 0 CHF positions.
|
||||||
|
// Paint first, then redraw once the reference data is in.
|
||||||
|
universe().then(() => activate(state.ui.tab, true)).catch(() => { /* the tab shows its own error */ });
|
||||||
|
|
||||||
|
let pending = null;
|
||||||
|
onChange((topic, source) => {
|
||||||
|
if (topic === "ui") return;
|
||||||
|
if (source && source === state.ui.tab) return; // the tab already updated itself
|
||||||
|
const tab = TABS.find((t) => t.id === state.ui.tab);
|
||||||
|
if (!tab?.deps.includes(topic)) return;
|
||||||
|
clearTimeout(pending);
|
||||||
|
pending = setTimeout(() => activate(state.ui.tab, true), 60);
|
||||||
|
});
|
||||||
47
quantcore/api/ui/js/runs.js
Normal file
47
quantcore/api/ui/js/runs.js
Normal file
@ -0,0 +1,47 @@
|
|||||||
|
/* Optimisation runs of this session, kept in memory so several tabs can refer to them.
|
||||||
|
*
|
||||||
|
* One of them is *selected*: the run the bench is showing and the one every "use this result" action
|
||||||
|
* works on. A new run selects itself; clicking a row in the history selects that one instead. Runs
|
||||||
|
* are addressed by `id`, never by their position -- `addRun` unshifts, so an index captured while
|
||||||
|
* rendering would point at a different run as soon as the next run arrives.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const runs = [];
|
||||||
|
let selectedId = null;
|
||||||
|
let nextId = 1;
|
||||||
|
|
||||||
|
export function addRun(entry) {
|
||||||
|
const run = { ...entry, id: nextId++, at: new Date() };
|
||||||
|
runs.unshift(run);
|
||||||
|
if (runs.length > 12) runs.pop();
|
||||||
|
selectedId = run.id;
|
||||||
|
return run;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function allRuns() { return runs; }
|
||||||
|
|
||||||
|
/* There is deliberately no `lastRun()`: "the newest run" and "the run the user is looking at" are
|
||||||
|
* different things once the history has more than one entry, and reaching for the newest was exactly
|
||||||
|
* the bug -- the holdings adopted were not the ones on screen. Use `selectedRun()`. */
|
||||||
|
|
||||||
|
/** The run on screen: what "this result" means in every tab. Falls back to the newest. */
|
||||||
|
export function selectedRun() {
|
||||||
|
return runs.find((r) => r.id === selectedId) || runs[0] || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function selectRun(id) {
|
||||||
|
if (runs.some((r) => r.id === id)) selectedId = id;
|
||||||
|
return selectedRun();
|
||||||
|
}
|
||||||
|
|
||||||
|
export function isSelected(id) {
|
||||||
|
const sel = selectedRun();
|
||||||
|
return !!sel && sel.id === id;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function clearRuns() { runs.length = 0; selectedId = null; }
|
||||||
|
|
||||||
|
/** Shares of a result, as a {symbol: shares} map. */
|
||||||
|
export function sharesOf(result) {
|
||||||
|
return Object.fromEntries((result?.positions || []).filter((p) => p.shares > 0).map((p) => [p.symbol, p.shares]));
|
||||||
|
}
|
||||||
179
quantcore/api/ui/js/sidebar.js
Normal file
179
quantcore/api/ui/js/sidebar.js
Normal file
@ -0,0 +1,179 @@
|
|||||||
|
/* The sidebar: the state every tab shares -- candidate stocks, the portfolio and the models. */
|
||||||
|
|
||||||
|
import {
|
||||||
|
api, button, changed, checkbox, el, field, fmt, input, intHoldings, onChange, resetState,
|
||||||
|
segmented, select, state, HORIZONS,
|
||||||
|
} from "./core.js";
|
||||||
|
import { info, topSymbols, universe } from "./universe.js";
|
||||||
|
|
||||||
|
const SCENARIO_MODELS = [
|
||||||
|
["gbm", "GBM (normal)"],
|
||||||
|
["empirical", "empirical innovations"],
|
||||||
|
["gaussian_exact", "exact Gaussian (no scenarios)"],
|
||||||
|
];
|
||||||
|
|
||||||
|
const SHOWN = ["selection", "universe", "portfolio", "model", "state"]; // topics the sidebar displays
|
||||||
|
let root = null;
|
||||||
|
|
||||||
|
export function buildSidebar(node) {
|
||||||
|
root = node;
|
||||||
|
draw(node);
|
||||||
|
// Only re-draw for changes made elsewhere: rebuilding the panel while someone is editing it
|
||||||
|
// would replace the control under the pointer (a button clicked right after typing is lost).
|
||||||
|
onChange((topic, source) => {
|
||||||
|
if (source !== "sidebar" && SHOWN.includes(topic)) draw(root);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Re-draw the sidebar itself (for edits that change which controls it shows). */
|
||||||
|
function refresh() { if (root) draw(root); }
|
||||||
|
|
||||||
|
const notify = (topic) => changed(topic, "sidebar");
|
||||||
|
|
||||||
|
function section(title, count, open, ...content) {
|
||||||
|
return el("details", { class: "section", open },
|
||||||
|
el("summary", {}, document.createTextNode(title), el("span", { class: "count", text: count ?? "" })),
|
||||||
|
el("div", {}, ...content));
|
||||||
|
}
|
||||||
|
|
||||||
|
function draw(node) {
|
||||||
|
node.replaceChildren(
|
||||||
|
candidatesSection(),
|
||||||
|
portfolioSection(),
|
||||||
|
modelSection(),
|
||||||
|
el("div", { class: "btnbar", style: { marginTop: "12px" } },
|
||||||
|
button("Reset all settings", () => { if (confirm("Reset every setting of the test bench?")) resetState(); }, "btn small")),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ candidates */
|
||||||
|
function candidatesSection() {
|
||||||
|
const u = state.universe;
|
||||||
|
const body = el("div");
|
||||||
|
const chips = el("div", { class: "chips", style: { marginTop: "8px" } });
|
||||||
|
|
||||||
|
const redraw = () => {
|
||||||
|
chips.replaceChildren();
|
||||||
|
if (!state.selection.length) {
|
||||||
|
chips.appendChild(el("span", { class: "hint", text: "no stocks selected - pick them in the Stocks tab or fill from the market" }));
|
||||||
|
}
|
||||||
|
for (const s of state.selection.slice(0, 60)) {
|
||||||
|
chips.appendChild(el("span", { class: "chip" }, document.createTextNode(s),
|
||||||
|
button("x", () => {
|
||||||
|
state.selection = state.selection.filter((x) => x !== s);
|
||||||
|
notify("selection"); redraw();
|
||||||
|
}, "")));
|
||||||
|
}
|
||||||
|
if (state.selection.length > 60) chips.appendChild(el("span", { class: "hint", text: `+ ${state.selection.length - 60} more` }));
|
||||||
|
};
|
||||||
|
redraw();
|
||||||
|
|
||||||
|
const fill = async () => {
|
||||||
|
const { rows } = await universe();
|
||||||
|
state.selection = topSymbols(rows, { region: u.region, sector: u.sector }, Number(u.top));
|
||||||
|
notify("selection");
|
||||||
|
refresh();
|
||||||
|
};
|
||||||
|
|
||||||
|
body.append(
|
||||||
|
el("div", { class: "row" },
|
||||||
|
field("Region", select([["", "world"], ["US", "US"], ["Europe", "Europe"]], u.region,
|
||||||
|
(v) => { u.region = v; notify("universe"); })),
|
||||||
|
field("Largest", input("number", u.top, (v) => { u.top = v || 30; notify("universe"); }, { min: 2, max: 500, step: 1 }))),
|
||||||
|
el("div", { class: "btnbar" },
|
||||||
|
button("Fill from market caps", fill, "btn small primary"),
|
||||||
|
button("Clear", () => { state.selection = []; notify("selection"); refresh(); }, "btn small")),
|
||||||
|
chips,
|
||||||
|
);
|
||||||
|
return section("Candidate stocks", `${state.selection.length}`, true, body);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ portfolio */
|
||||||
|
function portfolioSection() {
|
||||||
|
const p = state.portfolio;
|
||||||
|
const holdings = intHoldings(p.holdings);
|
||||||
|
const nHold = Object.keys(holdings).length;
|
||||||
|
const value = Object.entries(holdings).reduce((a, [s, n]) => a + (info(s)?.price || 0) * n, 0);
|
||||||
|
const body = el("div",
|
||||||
|
{},
|
||||||
|
field("Mode", segmented([["new", "New portfolio"], ["rebalance", "Rebalance"]], p.mode, (v) => {
|
||||||
|
p.mode = v; notify("portfolio"); refresh();
|
||||||
|
})),
|
||||||
|
field(p.mode === "new" ? "Cash to invest (CHF)" : "Free cash (CHF)",
|
||||||
|
input("number", p.cash, (v) => { p.cash = v ?? 0; notify("portfolio"); }, { min: 0, step: 10000 })),
|
||||||
|
el("div", { class: "btnbar", style: { marginBottom: "6px" } },
|
||||||
|
...[["1 m", 1e6], ["10 m", 1e7], ["50 m", 5e7]].map(([label, amount]) =>
|
||||||
|
button(label, () => { p.cash = amount; notify("portfolio"); refresh(); }, "btn small"))),
|
||||||
|
p.mode === "rebalance"
|
||||||
|
? el("div", { class: "note" },
|
||||||
|
`${nHold} holdings, ${fmt.compact(value)} CHF at the last close. Edit them in the Optimisation tab.`)
|
||||||
|
: el("div", { class: "note", text: "A new portfolio starts from cash; holdings are ignored." }),
|
||||||
|
);
|
||||||
|
return section("Portfolio", p.mode === "rebalance" ? `${nHold} positions` : fmt.compact(p.cash), true, body);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ models */
|
||||||
|
function modelSection() {
|
||||||
|
const m = state.model;
|
||||||
|
const set = (k, v) => { m[k] = v; notify("model"); };
|
||||||
|
const setShape = (k, v) => { m[k] = v; notify("model"); refresh(); };
|
||||||
|
const cov = m.covariance;
|
||||||
|
const exact = m.scenario_model === "gaussian_exact";
|
||||||
|
|
||||||
|
const covControls = cov.method === "ewm"
|
||||||
|
? field("EWM decay λ", input("number", cov.lam, (v) => { cov.lam = v; notify("model"); }, { min: 0.5, max: 0.9999, step: 0.005 }),
|
||||||
|
`≈ ${Math.round(3 / (1 - Number(cov.lam)))} effective observations`)
|
||||||
|
: field("Window (periods)", input("number", cov.window, (v) => { cov.window = v; notify("model"); }, { min: 10, step: 10 }));
|
||||||
|
|
||||||
|
return section("Models", m.scenario_model, true, el("div", {},
|
||||||
|
el("div", { class: "row" },
|
||||||
|
field("Frequency", select([["daily", "daily"], ["weekly", "weekly"]], m.freq, (v) => set("freq", v))),
|
||||||
|
field("Horizon", select(HORIZONS, m.horizon, (v) => set("horizon", v)))),
|
||||||
|
field("Stock-price model", select(SCENARIO_MODELS, m.scenario_model, (v) => setShape("scenario_model", v))),
|
||||||
|
exact ? el("div", { class: "note", text: "Exact Gaussian: mean-CVaR solved as the equivalent mean-variance problem by outer approximation - no sampling, no scenario noise." }) : null,
|
||||||
|
!exact ? el("div", { class: "row" },
|
||||||
|
field("Scenarios", input("number", m.n_scenarios, (v) => set("n_scenarios", v), { min: 256, step: 1024 }),
|
||||||
|
"rounded up to a power of two"),
|
||||||
|
field("Seed", input("number", m.seed, (v) => set("seed", v), { step: 1 }))) : null,
|
||||||
|
!exact ? el("div", { class: "row" },
|
||||||
|
field("Sampling", select([["sobol", "Sobol'"], ["mc", "plain MC"], ["antithetic", "antithetic"], ["lhs", "Latin hypercube"], ["balanced", "balanced (empirical)"]], m.sampling, (v) => set("sampling", v))),
|
||||||
|
field("Tail tilt", select([["auto", "auto"], ["market", "market"], ["adaptive", "adaptive"], ["off", "off"]], m.importance_sampling, (v) => set("importance_sampling", v)))) : null,
|
||||||
|
!exact ? checkbox("Moment matching (exact mean and covariance)", m.moment_match, (v) => set("moment_match", v)) : null,
|
||||||
|
el("div", { class: "row", style: { marginTop: "8px" } },
|
||||||
|
field("Covariance", select([["ewm", "EWM"], ["rolling", "rolling"]], cov.method, (v) => { cov.method = v; notify("model"); refresh(); })),
|
||||||
|
covControls),
|
||||||
|
field("Shrinkage", select([["auto", "auto (Schäfer-Strimmer)"], [0, "none"], [0.1, "0.10"], [0.25, "0.25"], [0.5, "0.50"]],
|
||||||
|
cov.shrinkage, (v) => { cov.shrinkage = v === "auto" ? "auto" : Number(v); notify("model"); })),
|
||||||
|
field("Expected returns", select([["black_litterman", "Black-Litterman"], ["historical", "historical mean"], ["zero", "zero drift"]],
|
||||||
|
m.expected_returns, (v) => setShape("expected_returns", v))),
|
||||||
|
m.expected_returns === "black_litterman"
|
||||||
|
? el("div", { class: "row" },
|
||||||
|
field("τ", input("number", m.bl_tau, (v) => set("bl_tau", v), { min: 0.001, step: 0.01 })),
|
||||||
|
field("History (years)", input("number", m.bl_years, (v) => set("bl_years", v), { min: 1, max: 20, step: 1 })))
|
||||||
|
: null,
|
||||||
|
field("Drift convention", select([["arithmetic", "arithmetic (Itô-corrected)"], ["log", "log"]], m.drift_convention, (v) => set("drift_convention", v))),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ health pill */
|
||||||
|
export async function refreshHealth(node) {
|
||||||
|
try {
|
||||||
|
const h = await api.get("/health");
|
||||||
|
const store = h.store || {};
|
||||||
|
const cuopt = h.cuopt || {};
|
||||||
|
node.replaceChildren(
|
||||||
|
pill("data", `${store.n_universe ?? "?"} stocks · ${fmt.date(store.last_price_date || "")}`, "ok"),
|
||||||
|
pill("cuOpt", cuopt.reachable ? (cuopt.version || "up") : "unreachable", cuopt.reachable ? "ok" : "warn"),
|
||||||
|
pill("HiGHS", "in-process", "ok"),
|
||||||
|
pill("base", h.base_currency || "CHF", "ok"),
|
||||||
|
el("span", { class: "pill" }, document.createTextNode(`v${h.version}`)),
|
||||||
|
);
|
||||||
|
} catch (err) {
|
||||||
|
node.replaceChildren(pill("api", String(err.message).slice(0, 60), "bad"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function pill(label, value, kind) {
|
||||||
|
return el("span", { class: "pill" }, el("span", { class: `dot ${kind}` }),
|
||||||
|
document.createTextNode(label + " "), el("b", { text: value }));
|
||||||
|
}
|
||||||
173
quantcore/api/ui/js/tabs/constraints.js
Normal file
173
quantcore/api/ui/js/tabs/constraints.js
Normal file
@ -0,0 +1,173 @@
|
|||||||
|
/* Tab 4 -- constraints and costs, with a feasibility check against the candidate set. */
|
||||||
|
|
||||||
|
import {
|
||||||
|
card, changed, chartBox, dataTable, el, empty, field, fmt, input, intHoldings, setChildren, state,
|
||||||
|
statTiles,
|
||||||
|
} from "../core.js";
|
||||||
|
import { barChart } from "../charts.js";
|
||||||
|
import { info } from "../universe.js";
|
||||||
|
|
||||||
|
export function render(node) {
|
||||||
|
node.replaceChildren();
|
||||||
|
if (!state.selection.length) { node.appendChild(empty("select candidate stocks in the Stocks tab first")); return; }
|
||||||
|
const derived = el("div", { id: "constraint-derived" });
|
||||||
|
const refresh = () => {
|
||||||
|
const live = document.getElementById("constraint-derived") || derived;
|
||||||
|
setChildren(live, feasibilityCard(), sectorCard());
|
||||||
|
};
|
||||||
|
refresh();
|
||||||
|
node.append(limitsCard(refresh), costsCard(refresh), derived);
|
||||||
|
}
|
||||||
|
|
||||||
|
function wealth() {
|
||||||
|
const h = intHoldings(state.portfolio.holdings);
|
||||||
|
const held = state.portfolio.mode === "rebalance"
|
||||||
|
? Object.entries(h).reduce((a, [s, n]) => a + (info(s)?.price || 0) * n, 0) : 0;
|
||||||
|
return Number(state.portfolio.cash) + held;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ limits */
|
||||||
|
function limitsCard(refresh) {
|
||||||
|
const c = state.constraints;
|
||||||
|
const set = (k) => (v) => { c[k] = v; changed("constraints", "constraints"); refresh(); };
|
||||||
|
const W = wealth();
|
||||||
|
const sectors = [...new Set(state.selection.map((s) => info(s)?.sector || "Unknown"))].sort();
|
||||||
|
|
||||||
|
const overrides = el("div");
|
||||||
|
const drawOverrides = () => {
|
||||||
|
overrides.replaceChildren(el("div", { class: "row" }, ...sectors.map((sec) =>
|
||||||
|
field(sec, input("number", c.sector_overrides?.[sec] ?? "", (v) => {
|
||||||
|
c.sector_overrides = { ...(c.sector_overrides || {}) };
|
||||||
|
if (v === null || v === "") delete c.sector_overrides[sec]; else c.sector_overrides[sec] = v;
|
||||||
|
changed("constraints", "constraints"); refresh();
|
||||||
|
}, { min: 0, max: 1, step: 0.05, placeholder: String(c.max_sector_weight) })))));
|
||||||
|
};
|
||||||
|
drawOverrides();
|
||||||
|
|
||||||
|
return card("Portfolio constraints", `fractions of wealth (${fmt.compact(W)} CHF)`, el("div", {},
|
||||||
|
el("div", { class: "row" },
|
||||||
|
field("Max weight per stock", input("number", c.max_weight, set("max_weight"), { min: 0.001, max: 1, step: 0.01 }),
|
||||||
|
`fraction of ${fmt.compact(W)} CHF per name`),
|
||||||
|
field("Max weight per sector", input("number", c.max_sector_weight, set("max_sector_weight"), { min: 0.01, max: 1, step: 0.05 }),
|
||||||
|
"applies to every GICS sector unless overridden below"),
|
||||||
|
field("Max positions", input("number", c.max_positions ?? "", set("max_positions"), { min: 1, step: 1, placeholder: "no limit" }),
|
||||||
|
"cardinality limit (adds binaries)"),
|
||||||
|
field("Min position value", input("number", c.min_position_value, set("min_position_value"), { min: 0, step: 1000 }),
|
||||||
|
"CHF; a position is either 0 or at least this")),
|
||||||
|
el("div", { class: "row" },
|
||||||
|
field("Min invested", input("number", c.min_invested, set("min_invested"), { min: 0, max: 1, step: 0.05 }),
|
||||||
|
"fraction of wealth that must be in stocks"),
|
||||||
|
field("Max invested", input("number", c.max_invested, set("max_invested"), { min: 0.01, max: 1, step: 0.05 }),
|
||||||
|
"the rest stays in cash"),
|
||||||
|
field("Max turnover", input("number", c.max_turnover ?? "", set("max_turnover"), { min: 0, step: 0.05, placeholder: "no limit" }),
|
||||||
|
"buys + sells as a fraction of wealth"),
|
||||||
|
field("Max transaction cost", input("number", c.max_transaction_cost ?? "", set("max_transaction_cost"),
|
||||||
|
{ min: 0, step: 500, placeholder: "no limit" }), "CHF")),
|
||||||
|
el("h3", { text: "Sector overrides", style: { marginTop: "10px" } }),
|
||||||
|
el("div", { class: "hint", text: "empty = the general sector cap" }),
|
||||||
|
overrides));
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ costs */
|
||||||
|
function costsCard(refresh) {
|
||||||
|
const k = state.costs;
|
||||||
|
const set = (key) => (v) => { k[key] = v ?? 0; changed("costs", "constraints"); refresh(); };
|
||||||
|
return card("Transaction costs", "charged on the traded value; they enter the objective and the cash balance",
|
||||||
|
el("div", { class: "row" },
|
||||||
|
field("Buy", input("number", k.proportional_buy, set("proportional_buy"), { min: 0, step: 0.0005 }), fmt.bp(k.proportional_buy)),
|
||||||
|
field("Sell", input("number", k.proportional_sell, set("proportional_sell"), { min: 0, step: 0.0005 }), fmt.bp(k.proportional_sell)),
|
||||||
|
field("Fixed per traded stock", input("number", k.fixed_per_trade, set("fixed_per_trade"), { min: 0, step: 5 }),
|
||||||
|
k.fixed_per_trade > 0 ? "CHF - adds one binary per stock" : "CHF (0 = no binaries)")));
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ feasibility */
|
||||||
|
function feasibilityCard() {
|
||||||
|
const c = state.constraints, W = wealth();
|
||||||
|
const sel = state.selection;
|
||||||
|
const bySector = {};
|
||||||
|
for (const s of sel) (bySector[info(s)?.sector || "Unknown"] ||= []).push(s);
|
||||||
|
const cap = (sec) => Number(c.sector_overrides?.[sec] ?? c.max_sector_weight);
|
||||||
|
|
||||||
|
const reachable = sel.filter((s) => (info(s)?.price || 0) <= c.max_weight * W);
|
||||||
|
const headroomBySector = Object.entries(bySector)
|
||||||
|
.map(([sec, list]) => Math.min(cap(sec), list.length * Number(c.max_weight)));
|
||||||
|
const headroom = headroomBySector.reduce((a, b) => a + b, 0);
|
||||||
|
const cardHeadroom = c.max_positions ? Number(c.max_positions) * Number(c.max_weight) : Infinity;
|
||||||
|
|
||||||
|
const checks = [
|
||||||
|
{ ok: sel.length * c.max_weight >= c.min_invested,
|
||||||
|
label: "Enough names for the minimum investment",
|
||||||
|
detail: `${sel.length} candidates × ${fmt.pct(c.max_weight, 0)} = ${fmt.pct(sel.length * c.max_weight, 0)} of wealth available, ${fmt.pct(c.min_invested, 0)} required` },
|
||||||
|
{ ok: headroom >= c.min_invested,
|
||||||
|
label: "Sector caps leave room for the minimum investment",
|
||||||
|
detail: `the sector caps allow at most ${fmt.pct(headroom, 0)} of wealth in total` },
|
||||||
|
{ ok: cardHeadroom >= c.min_invested,
|
||||||
|
label: "Cardinality limit leaves room",
|
||||||
|
detail: c.max_positions ? `${c.max_positions} positions × ${fmt.pct(c.max_weight, 0)} = ${fmt.pct(cardHeadroom, 0)}` : "no cardinality limit" },
|
||||||
|
{ ok: !c.min_position_value || c.min_position_value <= c.max_weight * W,
|
||||||
|
label: "Minimum position fits under the weight cap",
|
||||||
|
detail: `minimum ${fmt.compact(c.min_position_value)} CHF vs cap ${fmt.compact(c.max_weight * W)} CHF` },
|
||||||
|
{ ok: c.min_invested <= c.max_invested,
|
||||||
|
label: "Investment band is consistent",
|
||||||
|
detail: `${fmt.pct(c.min_invested, 0)} … ${fmt.pct(c.max_invested, 0)}` },
|
||||||
|
{ ok: reachable.length === sel.length,
|
||||||
|
label: "Every candidate is affordable as one share",
|
||||||
|
detail: reachable.length === sel.length ? "one share of each candidate fits under the weight cap"
|
||||||
|
: `${sel.length - reachable.length} stocks cost more per share than ${fmt.compact(c.max_weight * W)} CHF and can never be held` },
|
||||||
|
];
|
||||||
|
|
||||||
|
const failed = checks.filter((c2) => !c2.ok);
|
||||||
|
return card("Feasibility", failed.length ? `${failed.length} issue${failed.length > 1 ? "s" : ""} - the solver would return infeasible or ignore these names`
|
||||||
|
: "the constraints and the candidate set are consistent",
|
||||||
|
el("div", {},
|
||||||
|
statTiles([
|
||||||
|
{ label: "Candidates", value: fmt.int(sel.length), delta: `${Object.keys(bySector).length} sectors` },
|
||||||
|
{ label: "Wealth", value: `${fmt.compact(W)} CHF` },
|
||||||
|
{ label: "Cap per name", value: fmt.compact(c.max_weight * W), delta: fmt.pct(c.max_weight, 1) },
|
||||||
|
{ label: "Max reachable", value: fmt.pct(Math.min(headroom, cardHeadroom, sel.length * c.max_weight, c.max_invested), 0),
|
||||||
|
delta: "of wealth under all caps" },
|
||||||
|
{ label: "Min positions needed", value: fmt.int(Math.ceil(c.min_invested / c.max_weight)),
|
||||||
|
delta: `to reach ${fmt.pct(c.min_invested, 0)}` },
|
||||||
|
]),
|
||||||
|
el("div", { style: { marginTop: "10px" } }, dataTable([
|
||||||
|
{ key: "ok", label: "", render: (r) => el("span", { class: r.ok ? "pos" : "neg", text: r.ok ? "✓" : "✗" }) },
|
||||||
|
{ key: "label", label: "Check" },
|
||||||
|
{ key: "detail", label: "Detail", cls: "muted" },
|
||||||
|
], checks, { sortable: false }))));
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ sector composition */
|
||||||
|
function sectorCard() {
|
||||||
|
const c = state.constraints;
|
||||||
|
const sel = state.selection;
|
||||||
|
const bySector = {};
|
||||||
|
for (const s of sel) {
|
||||||
|
const sec = info(s)?.sector || "Unknown";
|
||||||
|
(bySector[sec] ||= { n: 0, cap: 0 });
|
||||||
|
bySector[sec].n += 1;
|
||||||
|
bySector[sec].cap += info(s)?.market_cap || 0;
|
||||||
|
}
|
||||||
|
const totalCap = Object.values(bySector).reduce((a, b) => a + b.cap, 0) || 1;
|
||||||
|
const items = Object.entries(bySector).sort((a, b) => b[1].cap - a[1].cap).map(([sec, v]) => ({
|
||||||
|
label: sec.length > 22 ? sec.slice(0, 21) + "…" : sec, values: [v.cap / totalCap],
|
||||||
|
}));
|
||||||
|
const box = chartBox();
|
||||||
|
const out = card("Sectors of the candidate set", "cap-weighted share of the selection against the cap that will be enforced",
|
||||||
|
el("div", {}, box,
|
||||||
|
dataTable([
|
||||||
|
{ key: "sector", label: "Sector" },
|
||||||
|
{ key: "n", label: "Stocks", num: true, fmt: (v) => fmt.int(v) },
|
||||||
|
{ key: "share", label: "Cap-weighted share", num: true, fmt: (v) => fmt.pct(v, 1) },
|
||||||
|
{ key: "limit", label: "Limit", num: true, fmt: (v) => fmt.pct(v, 0) },
|
||||||
|
{ key: "room", label: "Max under the caps", num: true, fmt: (v) => fmt.pct(v, 0),
|
||||||
|
title: "the smaller of the sector cap and (number of stocks × the per-stock cap)" },
|
||||||
|
], Object.entries(bySector).map(([sec, v]) => ({
|
||||||
|
sector: sec, n: v.n, share: v.cap / totalCap,
|
||||||
|
limit: Number(c.sector_overrides?.[sec] ?? c.max_sector_weight),
|
||||||
|
room: Math.min(Number(c.sector_overrides?.[sec] ?? c.max_sector_weight), v.n * Number(c.max_weight)),
|
||||||
|
})), { sortKey: "share", sortDir: -1 })));
|
||||||
|
barChart(box, { items, valueFormat: (v) => fmt.pct(v, 0), horizontal: true, rowHeight: 24, labelWidth: 150,
|
||||||
|
valueName: "cap-weighted share",
|
||||||
|
references: [{ value: Number(c.max_sector_weight), label: `sector cap ${fmt.pct(c.max_sector_weight, 0)}` }] });
|
||||||
|
return out;
|
||||||
|
}
|
||||||
511
quantcore/api/ui/js/tabs/optimize.js
Normal file
511
quantcore/api/ui/js/tabs/optimize.js
Normal file
@ -0,0 +1,511 @@
|
|||||||
|
/* Tab 5 -- the optimisation itself: inputs, the proposal, and before/after for a rebalancing. */
|
||||||
|
|
||||||
|
import {
|
||||||
|
api, button, card, changed, chartBox, constraintPayload, dataTable, el, empty, field, fmt, input,
|
||||||
|
intHoldings, kv, live, optimizePayload, segmented, select, setChildren, state, statTiles, warnBox,
|
||||||
|
withBusy,
|
||||||
|
} from "../core.js";
|
||||||
|
import { barChart, scatterChart } from "../charts.js";
|
||||||
|
import { info } from "../universe.js";
|
||||||
|
import { addRun, allRuns, clearRuns, isSelected, selectRun, selectedRun, sharesOf } from "../runs.js";
|
||||||
|
|
||||||
|
/** The result on screen. It is the *selected* run, so every "use this result" action and the
|
||||||
|
* history table agree on which one that is -- see js/runs.js. */
|
||||||
|
const shown = () => selectedRun()?.result || null;
|
||||||
|
|
||||||
|
export function render(node) {
|
||||||
|
node.replaceChildren();
|
||||||
|
if (!state.selection.length) { node.appendChild(empty("select candidate stocks in the Stocks tab first")); return; }
|
||||||
|
const rerender = () => render(node);
|
||||||
|
const out = el("div", { id: "opt-result" });
|
||||||
|
node.append(inputsCard(out, rerender), out, historyCard(rerender));
|
||||||
|
const current = shown();
|
||||||
|
if (current) drawResult(out, current);
|
||||||
|
else out.appendChild(empty("run the optimisation to see the proposal"));
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ inputs */
|
||||||
|
function inputsCard(out, rerender) {
|
||||||
|
const o = state.opt, p = state.portfolio;
|
||||||
|
// A plain value edit must not rebuild the card: the change event fires on blur, so rebuilding
|
||||||
|
// here would replace the button the user is clicking and swallow the click.
|
||||||
|
const set = (k) => (v) => { o[k] = v; changed("opt", "optimize"); };
|
||||||
|
const setShape = (k) => (v) => { o[k] = v; changed("opt", "optimize"); rerender(); };
|
||||||
|
let runBtn, sweepBtn;
|
||||||
|
|
||||||
|
const run = async (extra = {}, label = null) => {
|
||||||
|
const panel = live("opt-result", out);
|
||||||
|
await withBusy(runBtn, panel, async () => {
|
||||||
|
const body = optimizePayload(extra);
|
||||||
|
const t0 = performance.now();
|
||||||
|
const abort = new AbortController();
|
||||||
|
const stop = showProgress(panel, body, "", () => abort.abort());
|
||||||
|
let result;
|
||||||
|
try {
|
||||||
|
result = await api.post("/optimize", body, { signal: abort.signal });
|
||||||
|
} catch (err) {
|
||||||
|
if (err.name === "AbortError") { cancelled(live("opt-result", panel)); return; }
|
||||||
|
throw err;
|
||||||
|
} finally { stop(); }
|
||||||
|
addRun({ label: label || describeRun(body), request: body, result, wall: (performance.now() - t0) / 1000 });
|
||||||
|
drawResult(live("opt-result", panel), result);
|
||||||
|
rerender();
|
||||||
|
}).catch(() => {});
|
||||||
|
};
|
||||||
|
|
||||||
|
// The sweep traces the frontier along the parameter the *selected* objective is steered by. It
|
||||||
|
// used to force objective='max_return' and vary the risk limit whatever was selected, so a sweep
|
||||||
|
// started under "minimise CVaR" silently ran five constrained max-return problems and every one
|
||||||
|
// of them came back sitting on its CVaR limit.
|
||||||
|
const sweepPlan = () => {
|
||||||
|
if (o.objective === "min_cvar") {
|
||||||
|
const base = Number(o.target_return);
|
||||||
|
if (!Number.isFinite(base) || base <= 0) return null;
|
||||||
|
return {
|
||||||
|
what: "the target return",
|
||||||
|
steps: [0.25, 0.5, 1, 1.5, 2].map((k) => ({
|
||||||
|
extra: { target_return: round4(base * k) },
|
||||||
|
label: `E[r] ≥ ${fmt.pct(base * k, 3)}`,
|
||||||
|
})),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (o.objective === "utility") {
|
||||||
|
const base = Number(o.risk_aversion) || 1;
|
||||||
|
return {
|
||||||
|
what: "the risk aversion λ",
|
||||||
|
steps: [0.25, 0.5, 1, 2, 4].map((k) => ({
|
||||||
|
extra: { risk_aversion: round4(base * k) },
|
||||||
|
label: `λ = ${round4(base * k)}`,
|
||||||
|
})),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
what: "the risk limit",
|
||||||
|
steps: [0.01, 0.02, 0.03, 0.05, 0.08].map((f) => ({
|
||||||
|
extra: { cvar_limit_fraction: f, cvar_limit: null },
|
||||||
|
label: `CVaR ≤ ${fmt.pct(f, 0)}`,
|
||||||
|
})),
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
const sweep = async () => {
|
||||||
|
const plan = sweepPlan();
|
||||||
|
if (!plan) { alert("set a target return first - the sweep varies it around the value you entered"); return; }
|
||||||
|
await withBusy(sweepBtn, live("opt-result", out), async () => {
|
||||||
|
for (const step of plan.steps) {
|
||||||
|
const body = optimizePayload(step.extra);
|
||||||
|
const t0 = performance.now();
|
||||||
|
const abort = new AbortController();
|
||||||
|
const stop = showProgress(live("opt-result", out), body, step.label, () => abort.abort());
|
||||||
|
let result;
|
||||||
|
try {
|
||||||
|
result = await api.post("/optimize", body, { signal: abort.signal });
|
||||||
|
} catch (err) {
|
||||||
|
if (err.name === "AbortError") { cancelled(live("opt-result", out)); break; }
|
||||||
|
throw err;
|
||||||
|
} finally { stop(); }
|
||||||
|
addRun({ label: step.label, request: body, result, wall: (performance.now() - t0) / 1000 });
|
||||||
|
drawResult(live("opt-result", out), result);
|
||||||
|
}
|
||||||
|
rerender();
|
||||||
|
}).catch(() => {});
|
||||||
|
};
|
||||||
|
|
||||||
|
runBtn = button("Run optimisation", () => run(), "btn primary");
|
||||||
|
const sweepWhat = o.objective === "min_cvar" ? "target return"
|
||||||
|
: o.objective === "utility" ? "\u03bb" : "risk limit";
|
||||||
|
sweepBtn = button(`Sweep the ${sweepWhat} (5 runs)`, sweep, "btn",
|
||||||
|
`five runs varying ${sweepWhat} around the current value, keeping the objective`);
|
||||||
|
|
||||||
|
// the hint under the limit is updated in place (see `set` above)
|
||||||
|
const limitHint = el("span", { class: "hint", text: limitHintText() });
|
||||||
|
const limitField = (label, control) => el("div", { class: "field" }, el("label", { text: label }), control, limitHint);
|
||||||
|
const onLimit = (k) => (v) => { set(k)(v); limitHint.textContent = limitHintText(); };
|
||||||
|
const limitControls = o.objective === "max_return"
|
||||||
|
? el("div", { class: "row" },
|
||||||
|
field("CVaR limit", segmented([["fraction", "% of wealth"], ["money", "CHF"]], o.cvar_mode, setShape("cvar_mode"))),
|
||||||
|
o.cvar_mode === "fraction"
|
||||||
|
// entered in percent, because the control above says "% of wealth"
|
||||||
|
? limitField("Limit (% of wealth)", input("number", round4((o.cvar_limit_fraction || 0) * 100),
|
||||||
|
(v) => onLimit("cvar_limit_fraction")((v ?? 0) / 100),
|
||||||
|
{ min: 0.1, max: 99, step: 0.5 }))
|
||||||
|
: limitField("Limit (CHF)", input("number", o.cvar_limit, onLimit("cvar_limit"), { min: 1, step: 1000 })))
|
||||||
|
: o.objective === "min_cvar"
|
||||||
|
? field("Target return over the horizon", input("number", o.target_return ?? "", set("target_return"), { step: 0.001, placeholder: "none" }),
|
||||||
|
"fraction of wealth, e.g. 0.002")
|
||||||
|
: field("Risk aversion λ", input("number", o.risk_aversion, set("risk_aversion"), { min: 0, step: 0.01 }),
|
||||||
|
"maximise E[P&L] − λ · CVaR; both are money, so λ is the francs of expected return you "
|
||||||
|
+ "give up per franc of CVaR — over a week the interesting range is around 0.01–0.05, "
|
||||||
|
+ "and a large λ keeps everything in cash");
|
||||||
|
|
||||||
|
return card("Optimisation", "mean-CVaR in integer shares; the model panel on the left decides how the scenarios are made",
|
||||||
|
el("div", {},
|
||||||
|
el("div", { class: "row" },
|
||||||
|
field("Mode", segmented([["new", "New portfolio"], ["rebalance", "Rebalance"]], p.mode, (v) => {
|
||||||
|
p.mode = v; changed("portfolio", "optimize"); rerender(); })),
|
||||||
|
field("Objective", select([["max_return", "maximise expected return"], ["min_cvar", "minimise CVaR"],
|
||||||
|
["utility", "expected return − λ · CVaR"]], o.objective, setShape("objective"))),
|
||||||
|
field("Confidence β", select([[0.9, "90 %"], [0.95, "95 %"], [0.99, "99 %"]], o.confidence, (v) => set("confidence")(Number(v)))),
|
||||||
|
field("Solver", select([["auto", "auto (cuOpt, then HiGHS)"], ["cuopt", "cuOpt (GPU)"], ["highs", "HiGHS (CPU)"]], o.solver, set("solver"))),
|
||||||
|
field("Time limit (s)", input("number", o.time_limit ?? "", set("time_limit"), { min: 1, step: 10, placeholder: "no limit" }),
|
||||||
|
o.time_limit ? "budget for the whole run; the solver returns its best solution at that point"
|
||||||
|
: "empty: HiGHS runs until proven optimal - a large problem can take hours")),
|
||||||
|
limitControls,
|
||||||
|
el("div", { class: "row" },
|
||||||
|
field("Risk measured on", select([["model", "scenarios incl. the return forecast"],
|
||||||
|
["zero", "centred scenarios (no drift)"]], o.risk_drift, set("risk_drift")),
|
||||||
|
o.risk_drift === "model"
|
||||||
|
? "over a long horizon a large forecast can push the whole tail into profit"
|
||||||
|
: "the risk figure does not depend on the return forecast"),
|
||||||
|
field("Tail rows", select([["auto", "auto"], ["on", "always"], ["off", "never"]], o.row_generation, set("row_generation")),
|
||||||
|
"solve on the worst scenarios only, then verify"),
|
||||||
|
el("div", { class: "field" },
|
||||||
|
el("label", { text: "Scenario count" }),
|
||||||
|
segmented([[true, "automatic"], [false, `fixed (${fmt.int(state.model.n_scenarios)})`]], o.auto_scenarios,
|
||||||
|
(v) => setShape("auto_scenarios")(v === "true" || v === true))),
|
||||||
|
el("div", { class: "field auto" }, el("label", { text: " " }), el("div", { class: "btnbar" }, runBtn, sweepBtn))),
|
||||||
|
constraintStrip(),
|
||||||
|
p.mode === "rebalance" ? holdingsEditor(rerender) : null));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The constraints and costs this run will send -- edited in the Constraints tab, shown here so it
|
||||||
|
is never a guess which limits are in force. */
|
||||||
|
function constraintStrip() {
|
||||||
|
const c = constraintPayload(), k = state.costs;
|
||||||
|
const sector = typeof c.max_sector_weight === "object"
|
||||||
|
? `${fmt.pct(c.max_sector_weight["*"], 0)} per sector (${Object.keys(c.max_sector_weight).length - 1} overridden)`
|
||||||
|
: `${fmt.pct(c.max_sector_weight, 0)} per sector`;
|
||||||
|
const parts = [
|
||||||
|
`${fmt.pct(c.max_weight, 0)} per stock`, sector,
|
||||||
|
c.max_positions ? `at most ${c.max_positions} positions` : "no cardinality limit",
|
||||||
|
c.min_position_value ? `min position ${fmt.compact(c.min_position_value)} CHF` : null,
|
||||||
|
`invested ${fmt.pct(c.min_invested, 0)} – ${fmt.pct(c.max_invested, 0)}`,
|
||||||
|
c.max_turnover !== null ? `turnover ≤ ${fmt.pct(c.max_turnover, 0)}` : null,
|
||||||
|
c.max_transaction_cost !== null ? `cost ≤ ${fmt.compact(c.max_transaction_cost)} CHF` : null,
|
||||||
|
`costs ${fmt.bp(k.proportional_buy)} buy / ${fmt.bp(k.proportional_sell)} sell`,
|
||||||
|
Number(k.fixed_per_trade) ? `+ ${fmt.money(k.fixed_per_trade)} CHF per traded stock` : null,
|
||||||
|
].filter(Boolean);
|
||||||
|
return el("div", { class: "note", style: { marginTop: "10px" } },
|
||||||
|
el("b", { text: "Constraints in force: " }), document.createTextNode(parts.join(" · ") + " "),
|
||||||
|
button("edit", () => document.dispatchEvent(new CustomEvent("quantcore:tab", { detail: "constraints" })), "btn small"));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A live "solving …" panel with a cancel button; returns the function that stops the timer. */
|
||||||
|
function showProgress(out, body, extra = "", onCancel = null) {
|
||||||
|
const started = performance.now();
|
||||||
|
const secs = el("b", { text: "0 s" });
|
||||||
|
const line = el("div", { class: "empty" }, el("span", { class: "spinner" }),
|
||||||
|
document.createTextNode(" solving … "), secs,
|
||||||
|
onCancel ? document.createTextNode(" ") : null,
|
||||||
|
onCancel ? button("Cancel", onCancel, "btn small") : null);
|
||||||
|
const limit = body.time_limit ? `, time limit ${fmt.int(body.time_limit)} s` : ", no time limit";
|
||||||
|
const detail = el("div", { class: "hint", style: { marginTop: "6px" } },
|
||||||
|
`${body.scenario_model} · ${body.n_scenarios ? fmt.int(body.n_scenarios) + " scenarios" : "automatic scenario count"} · `
|
||||||
|
+ `${body.symbols?.length ?? body.max_assets} candidates · solver ${body.solver}${limit}${extra ? " · " + extra : ""}`);
|
||||||
|
// the service solves one problem at a time (the MILP holds the interpreter), so say so
|
||||||
|
const queued = el("div", { class: "hint" },
|
||||||
|
"The service answers one request at a time while a solve runs - the other tabs wait until this finishes.");
|
||||||
|
out.replaceChildren(line, detail, queued);
|
||||||
|
const t = setInterval(() => { secs.textContent = `${((performance.now() - started) / 1000).toFixed(0)} s`; }, 500);
|
||||||
|
return () => clearInterval(t);
|
||||||
|
}
|
||||||
|
|
||||||
|
function cancelled(panel) {
|
||||||
|
setChildren(panel, el("div", { class: "warn-box", text:
|
||||||
|
"Run cancelled. The server finishes the solve it already started (it stops at the time limit); "
|
||||||
|
+ "the next run is unaffected." }));
|
||||||
|
}
|
||||||
|
|
||||||
|
const round4 = (v) => Math.round(v * 1e4) / 1e4; // 0.045 * 100 must read as 4.5, not 4.499999
|
||||||
|
|
||||||
|
function limitHintText() {
|
||||||
|
const o = state.opt;
|
||||||
|
return o.cvar_mode === "fraction"
|
||||||
|
? `the expected shortfall may not exceed ${fmt.compact((o.cvar_limit_fraction || 0) * wealth())} CHF `
|
||||||
|
+ `over one ${state.model.horizon}`
|
||||||
|
: `${fmt.pct((o.cvar_limit || 0) / wealth(), 1)} of wealth over one ${state.model.horizon}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function wealth() {
|
||||||
|
const h = intHoldings(state.portfolio.holdings);
|
||||||
|
const held = state.portfolio.mode === "rebalance"
|
||||||
|
? Object.entries(h).reduce((a, [s, n]) => a + (info(s)?.price || 0) * n, 0) : 0;
|
||||||
|
return Number(state.portfolio.cash) + held;
|
||||||
|
}
|
||||||
|
|
||||||
|
function describeRun(body) {
|
||||||
|
const risk = body.objective === "min_cvar"
|
||||||
|
? `min CVaR, E[r] \u2265 ${fmt.pct(body.target_return ?? 0, 3)}`
|
||||||
|
: body.objective === "utility" ? `E[P&L] \u2212 ${round4(body.risk_aversion)} \u00b7 CVaR`
|
||||||
|
: body.cvar_limit_fraction ? `CVaR \u2264 ${fmt.pct(body.cvar_limit_fraction, 1)}`
|
||||||
|
: body.cvar_limit ? `CVaR \u2264 ${fmt.compact(body.cvar_limit)}` : body.objective;
|
||||||
|
return `${body.mode} · ${body.scenario_model} · ${risk} · β ${fmt.pct(body.confidence, 0)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Cash is money: two decimals, not the full float that a valuation produces. */
|
||||||
|
function round2(x) { return Math.round(x * 100) / 100; }
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ holdings */
|
||||||
|
function holdingsEditor(rerender) {
|
||||||
|
const p = state.portfolio;
|
||||||
|
const holdings = intHoldings(p.holdings);
|
||||||
|
const rows = Object.entries(holdings).map(([symbol, shares]) => ({
|
||||||
|
symbol, name: info(symbol)?.name || symbol, shares, price: info(symbol)?.price,
|
||||||
|
value: (info(symbol)?.price || 0) * shares,
|
||||||
|
}));
|
||||||
|
const total = rows.reduce((a, r) => a + r.value, 0);
|
||||||
|
|
||||||
|
const addSel = select(["", ...state.selection.filter((s) => !(s in holdings))].map((s) => [s, s || "add a stock …"]), "",
|
||||||
|
(v) => { if (!v) return; p.holdings = { ...holdings, [v]: 1 }; changed("portfolio", "optimize"); rerender(); });
|
||||||
|
|
||||||
|
return el("div", { style: { marginTop: "12px" } },
|
||||||
|
el("h3", { text: "Current holdings" }),
|
||||||
|
el("div", { class: "hint", text: `${rows.length} positions, ${fmt.compact(total)} CHF at the last close, plus ${fmt.compact(p.cash)} CHF cash` }),
|
||||||
|
el("div", { class: "btnbar", style: { margin: "7px 0" } },
|
||||||
|
button("Use the shown result", () => {
|
||||||
|
// the run on screen, not the newest one: with several runs in the history those differ, and
|
||||||
|
// the weights that get adopted have to be the ones the user is looking at
|
||||||
|
const r = selectedRun();
|
||||||
|
if (!r) { alert("no optimisation has run yet"); return; }
|
||||||
|
p.holdings = sharesOf(r.result);
|
||||||
|
// the proposal spent the cash: carry over what it left, or the portfolio would be the new
|
||||||
|
// positions *plus* the money already invested in them and every weight would come out low
|
||||||
|
p.cash = round2(Math.max(0, Number(r.result.cash_end ?? p.cash)));
|
||||||
|
p.mode = "rebalance"; changed("portfolio", "optimize"); rerender();
|
||||||
|
}, "btn small", selectedRun() ? `takes the ${selectedRun().result.n_positions} positions of "${selectedRun().label}"` : "run an optimisation first"),
|
||||||
|
selectedRun() ? el("span", { class: "hint", text: `shown: ${selectedRun().label}` }) : null,
|
||||||
|
button("Equal weight from cash", () => {
|
||||||
|
const cash = Number(p.cash);
|
||||||
|
const each = cash / Math.max(state.selection.length, 1);
|
||||||
|
p.holdings = Object.fromEntries(state.selection.map((s) => [s, Math.floor(each / (info(s)?.price || 1))])
|
||||||
|
.filter(([, n]) => n > 0));
|
||||||
|
// likewise: the cash has been turned into shares, only the rounding remainder is left
|
||||||
|
const spent = Object.entries(p.holdings).reduce((a, [s, n]) => a + (info(s)?.price || 0) * n, 0);
|
||||||
|
p.cash = round2(Math.max(0, cash - spent));
|
||||||
|
changed("portfolio", "optimize"); rerender();
|
||||||
|
}, "btn small", "buys equal amounts of the selected stocks and leaves the rounding remainder as cash"),
|
||||||
|
button("Clear", () => { p.holdings = {}; changed("portfolio", "optimize"); rerender(); }, "btn small"),
|
||||||
|
addSel),
|
||||||
|
rows.length ? dataTable([
|
||||||
|
{ key: "symbol", label: "Symbol", cls: "sym" },
|
||||||
|
{ key: "name", label: "Name", cls: "name" },
|
||||||
|
{ key: "price", label: "Price", num: true, fmt: (v) => fmt.money2(v) },
|
||||||
|
{ key: "shares", label: "Shares", num: true, render: (r) => {
|
||||||
|
const box = el("input", { type: "number", value: r.shares, min: 0, step: 1, style: { width: "92px", textAlign: "right" } });
|
||||||
|
box.addEventListener("change", () => {
|
||||||
|
const next = { ...intHoldings(p.holdings) };
|
||||||
|
const n = Math.max(0, Math.round(Number(box.value) || 0));
|
||||||
|
if (n === 0) delete next[r.symbol]; else next[r.symbol] = n;
|
||||||
|
p.holdings = next; changed("portfolio", "optimize");
|
||||||
|
});
|
||||||
|
return box;
|
||||||
|
} },
|
||||||
|
{ key: "value", label: "Value (CHF)", num: true, fmt: (v) => fmt.money(v) },
|
||||||
|
{ key: "w", label: "Weight", num: true, render: (r) => document.createTextNode(fmt.pct(r.value / (total + Number(p.cash)), 1)) },
|
||||||
|
{ key: "del", label: "", render: (r) => button("remove", () => {
|
||||||
|
const next = { ...intHoldings(p.holdings) }; delete next[r.symbol];
|
||||||
|
p.holdings = next; changed("portfolio", "optimize"); rerender();
|
||||||
|
}, "btn small") },
|
||||||
|
], rows, { sortKey: "value", sortDir: -1 }) : el("div", { class: "note", text: "no holdings - add stocks or start from a previous result" }));
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ result */
|
||||||
|
function drawResult(out, r) {
|
||||||
|
const before = r.before;
|
||||||
|
const ok = r.status === "Optimal" || r.status === "FeasibleFound";
|
||||||
|
const tiles = [
|
||||||
|
{ label: "Status", value: r.status, delta: `${r.solver} · ${fmt.secs(r.timings?.solve_s)}`,
|
||||||
|
deltaClass: ok ? "muted" : "neg" },
|
||||||
|
{ label: "Expected P&L", value: fmt.money(r.expected_pnl), delta: `${fmt.pctSigned(r.expected_return, 2)} of wealth`,
|
||||||
|
deltaClass: r.expected_pnl >= 0 ? "pos" : "neg" },
|
||||||
|
{ label: `CVaR ${fmt.pct(r.confidence, 0)}`, value: fmt.money(r.cvar),
|
||||||
|
delta: r.cvar_limit ? `limit ${fmt.money(r.cvar_limit)}` : "no limit",
|
||||||
|
deltaClass: r.cvar_limit && r.cvar > r.cvar_limit * 1.001 ? "neg" : "muted" },
|
||||||
|
{ label: `VaR ${fmt.pct(r.confidence, 0)}`, value: fmt.money(r.var) },
|
||||||
|
{ label: "Invested", value: fmt.money(r.invested), delta: `${fmt.pct(r.invested / r.wealth, 1)} of ${fmt.compact(r.wealth)}` },
|
||||||
|
{ label: "Positions", value: fmt.int(r.n_positions), delta: `of ${fmt.int(r.n_candidates)} candidates` },
|
||||||
|
{ label: "Turnover", value: fmt.pct(r.turnover, 1), delta: `cost ${fmt.money(r.total_cost)}` },
|
||||||
|
{ label: "Reproducible", value: r.reproducibility, delta: r.scenario_hash ? `set ${r.scenario_hash.slice(0, 8)}` : "no scenarios",
|
||||||
|
deltaClass: r.reproducibility === "exact" ? "pos" : "muted" },
|
||||||
|
];
|
||||||
|
|
||||||
|
const weightChart = chartBox(), sectorChart = chartBox();
|
||||||
|
const positions = r.positions || [];
|
||||||
|
const parts = [warnBox(r.warnings), statTiles(tiles)];
|
||||||
|
|
||||||
|
if (before) parts.push(beforeAfterCard(r, before));
|
||||||
|
|
||||||
|
parts.push(el("div", { class: "grid two", style: { marginTop: "12px" } },
|
||||||
|
card("Portfolio weights", before ? "before and after the rebalancing" : "share of wealth per stock", weightChart),
|
||||||
|
card("Sector exposure", "against the limit that was enforced", sectorChart)));
|
||||||
|
|
||||||
|
parts.push(card("Positions", before ? "shares before and after, with the trades" : "the proposed portfolio",
|
||||||
|
positionsTable(positions, !!before, r.wealth)));
|
||||||
|
|
||||||
|
parts.push(provenanceCard(r));
|
||||||
|
setChildren(out, ...parts);
|
||||||
|
|
||||||
|
const held = positions.filter((p) => p.shares > 0 || p.shares_before > 0)
|
||||||
|
.sort((a, b) => b.value - a.value).slice(0, 25);
|
||||||
|
barChart(weightChart, {
|
||||||
|
items: held.map((p) => ({ label: p.symbol, values: before ? [p.shares_before * p.price / r.wealth, p.weight] : [p.weight] })),
|
||||||
|
groups: before ? ["before", "after"] : [""],
|
||||||
|
valueFormat: (v) => fmt.pct(v, 1), horizontal: true, rowHeight: before ? 24 : 20, labelWidth: 84,
|
||||||
|
references: [{ value: state.constraints.max_weight, label: "cap" }], valueName: "weight",
|
||||||
|
});
|
||||||
|
const sectors = Object.entries(r.sector_exposure || {}).sort((a, b) => b[1] - a[1]);
|
||||||
|
const cap = Number(state.constraints.max_sector_weight);
|
||||||
|
barChart(sectorChart, {
|
||||||
|
items: sectors.map(([sec, v]) => ({ label: sec.length > 20 ? sec.slice(0, 19) + "…" : sec,
|
||||||
|
values: before ? [before.sector_exposure?.[sec] || 0, v] : [v] })),
|
||||||
|
groups: before ? ["before", "after"] : [""], valueName: "exposure",
|
||||||
|
valueFormat: (v) => fmt.pct(v, 0), horizontal: true, rowHeight: before ? 28 : 22, labelWidth: 140,
|
||||||
|
references: [{ value: cap, label: `sector cap ${fmt.pct(cap, 0)}` }],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function beforeAfterCard(r, before) {
|
||||||
|
const rows = [
|
||||||
|
{ k: "Invested (CHF)", a: before.invested, b: r.invested, f: fmt.money },
|
||||||
|
{ k: "Cash (CHF)", a: before.cash, b: r.cash_end, f: fmt.money },
|
||||||
|
{ k: "Positions", a: before.n_positions, b: r.n_positions, f: fmt.int },
|
||||||
|
{ k: "Expected P&L (CHF)", a: before.expected_pnl, b: r.expected_pnl, f: fmt.money },
|
||||||
|
{ k: "Expected return of wealth", a: before.expected_return, b: r.expected_return, f: (v) => fmt.pctSigned(v, 3) },
|
||||||
|
{ k: `VaR ${fmt.pct(r.confidence, 0)} (CHF)`, a: before.var, b: r.var, f: fmt.money },
|
||||||
|
{ k: `CVaR ${fmt.pct(r.confidence, 0)} (CHF)`, a: before.cvar, b: r.cvar, f: fmt.money },
|
||||||
|
{ k: "CVaR of wealth", a: before.cvar_fraction, b: r.cvar / r.wealth, f: (v) => fmt.pct(v, 2) },
|
||||||
|
].map((row) => ({ ...row, d: row.b - row.a }));
|
||||||
|
|
||||||
|
return card("Before and after", "the starting portfolio measured on the same scenarios as the proposal",
|
||||||
|
dataTable([
|
||||||
|
{ key: "k", label: "" },
|
||||||
|
{ key: "a", label: "Before", num: true, fmt: (v, r2) => r2.f(v) },
|
||||||
|
{ key: "b", label: "After", num: true, fmt: (v, r2) => r2.f(v) },
|
||||||
|
{ key: "d", label: "Change", num: true, fmt: (v, r2) => (r2.f === fmt.int ? fmt.signed(v) : r2.f(v)),
|
||||||
|
cls: (r2) => (Math.abs(r2.d) < 1e-9 ? "muted" : r2.d > 0 ? "pos" : "neg") },
|
||||||
|
], rows, { sortable: false }));
|
||||||
|
}
|
||||||
|
|
||||||
|
function positionsTable(positions, rebalancing, wealthValue) {
|
||||||
|
const cols = [
|
||||||
|
{ key: "symbol", label: "Symbol", cls: "sym" },
|
||||||
|
{ key: "name", label: "Name", cls: "name" },
|
||||||
|
{ key: "sector", label: "Sector" },
|
||||||
|
{ key: "price", label: "Price", num: true, fmt: (v) => fmt.money2(v) },
|
||||||
|
];
|
||||||
|
if (rebalancing) cols.push(
|
||||||
|
{ key: "shares_before", label: "Shares before", num: true, fmt: (v) => fmt.int(v) },
|
||||||
|
{ key: "shares", label: "Shares after", num: true, fmt: (v) => fmt.int(v) },
|
||||||
|
{ key: "trade", label: "Trade", num: true, fmt: (v) => fmt.signed(v),
|
||||||
|
cls: (r) => (r.trade > 0 ? "pos" : r.trade < 0 ? "neg" : "muted") },
|
||||||
|
{ key: "trade_value", label: "Traded (CHF)", num: true, fmt: (v) => fmt.signed(v) },
|
||||||
|
);
|
||||||
|
else cols.push({ key: "shares", label: "Shares", num: true, fmt: (v) => fmt.int(v) });
|
||||||
|
cols.push(
|
||||||
|
{ key: "value", label: "Value (CHF)", num: true, fmt: (v) => fmt.money(v) },
|
||||||
|
{ key: "weight", label: "Weight", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
{ key: "cost", label: "Cost (CHF)", num: true, fmt: (v) => fmt.money2(v) },
|
||||||
|
{ key: "expected_return_ann", label: "E[r] p.a.", num: true, fmt: (v) => fmt.pct(v, 1) },
|
||||||
|
{ key: "expected_return_horizon", label: "E[r] horizon", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
);
|
||||||
|
return dataTable(cols, positions, { sortKey: "value", sortDir: -1 });
|
||||||
|
}
|
||||||
|
|
||||||
|
function provenanceCard(r) {
|
||||||
|
const m = r.model || {}, s = r.solver_info || {};
|
||||||
|
const scen = m.scenarios || {};
|
||||||
|
const rg = s.row_generation;
|
||||||
|
return card("How this number was produced", "everything needed to reproduce the run",
|
||||||
|
el("div", { class: "grid three" },
|
||||||
|
el("div", {}, el("h3", { text: "Scenarios" }),
|
||||||
|
kv([
|
||||||
|
["model", scen.model], ["scenarios", fmt.int(scen.n_scenarios)], ["sampling", m.sampling?.sampling],
|
||||||
|
["moment matching", String(m.sampling?.moment_match)], ["tail tilt", m.sampling?.importance_sampling],
|
||||||
|
["effective sample", scen.effective_sample_size ? fmt.int(scen.effective_sample_size) : null],
|
||||||
|
["risk measured on", m.sampling?.risk_drift === "zero" ? "centred scenarios" : "scenarios incl. the forecast"],
|
||||||
|
["tilt alignment", Number.isFinite(s.tilt_alignment) ? fmt.num(s.tilt_alignment, 2) : null],
|
||||||
|
["scenario set", r.scenario_hash],
|
||||||
|
["reused from cache", r.solver_info?.scenario_cache ? (r.solver_info.scenario_cache.reused ? "yes" : "no") : null],
|
||||||
|
["as of", scen.as_of], ["seed", scen.seed],
|
||||||
|
])),
|
||||||
|
el("div", {}, el("h3", { text: "Solver" }),
|
||||||
|
kv([
|
||||||
|
["solver", r.solver], ["status", r.status], ["objective", Number.isFinite(s.objective) ? fmt.money2(s.objective) : null],
|
||||||
|
["MIP gap", Number.isFinite(s.mip_gap) ? s.mip_gap.toExponential(1) : s.mip_gap],
|
||||||
|
["variables", fmt.int(m.milp?.n_vars)],
|
||||||
|
["rows solved", rg ? `${fmt.int(rg.rows_final)} of ${fmt.int(scen.n_scenarios)}` : null],
|
||||||
|
["row-generation passes", rg?.iterations], ["cuts (Gaussian)", s.outer_approximation?.iterations],
|
||||||
|
["solve time", fmt.secs(s.solve_time_s)], ["total", fmt.secs(r.timings?.total_s)],
|
||||||
|
["reproducibility", r.reproducibility],
|
||||||
|
])),
|
||||||
|
el("div", {}, el("h3", { text: "Constraints used" }),
|
||||||
|
kv(constraintRows(m.constraints || {}, m.costs || {}))),
|
||||||
|
el("div", {}, el("h3", { text: "Expected returns" }),
|
||||||
|
kv([
|
||||||
|
["model", m.expected_returns?.type], ["market return p.a.", m.expected_returns?.market_return_ann ? fmt.pct(m.expected_returns.market_return_ann, 2) : null],
|
||||||
|
["δ", m.expected_returns?.delta ? fmt.num(m.expected_returns.delta, 2) : null],
|
||||||
|
["τ", m.expected_returns?.tau], ["views", m.expected_returns?.views?.length ?? 0],
|
||||||
|
["drift convention", m.sampling?.drift_convention],
|
||||||
|
["covariance", `${scen.covariance_model?.method ?? ""} ${scen.covariance_model?.lam ?? scen.covariance_model?.window ?? ""}`],
|
||||||
|
["shrinkage", scen.shrinkage_applied ? fmt.pct(scen.shrinkage_applied, 1) : null],
|
||||||
|
]))));
|
||||||
|
}
|
||||||
|
|
||||||
|
function constraintRows(c, k) {
|
||||||
|
const sector = c.max_sector_weight && typeof c.max_sector_weight === "object"
|
||||||
|
? Object.entries(c.max_sector_weight).map(([sec, v]) => `${sec === "*" ? "all" : sec} ${fmt.pct(v, 0)}`).join(", ")
|
||||||
|
: fmt.pct(c.max_sector_weight, 0);
|
||||||
|
return [
|
||||||
|
["max per stock", fmt.pct(c.max_weight, 1)],
|
||||||
|
["max per sector", sector],
|
||||||
|
["max positions", c.max_positions ?? "no limit"],
|
||||||
|
["min position", c.min_position_value ? `${fmt.money(c.min_position_value)} CHF` : "none"],
|
||||||
|
["invested", `${fmt.pct(c.min_invested, 0)} – ${fmt.pct(c.max_invested, 0)}`],
|
||||||
|
["max turnover", c.max_turnover === null || c.max_turnover === undefined ? "no limit" : fmt.pct(c.max_turnover, 0)],
|
||||||
|
["max cost", c.max_transaction_cost === null || c.max_transaction_cost === undefined
|
||||||
|
? "no limit" : `${fmt.money(c.max_transaction_cost)} CHF`],
|
||||||
|
["trading cost", `${fmt.bp(k.proportional_buy)} buy / ${fmt.bp(k.proportional_sell)} sell`],
|
||||||
|
["fixed per trade", k.fixed_per_trade ? `${fmt.money(k.fixed_per_trade)} CHF` : "none"],
|
||||||
|
];
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ history */
|
||||||
|
function historyCard(rerender) {
|
||||||
|
const runs = allRuns();
|
||||||
|
if (!runs.length) return el("span");
|
||||||
|
const rows = runs.map((run) => ({
|
||||||
|
id: run.id, label: run.label, status: run.result.status,
|
||||||
|
pnl: run.result.expected_pnl, ret: run.result.expected_return, cvar: run.result.cvar,
|
||||||
|
cvar_w: run.result.cvar / run.result.wealth, n: run.result.n_positions,
|
||||||
|
turnover: run.result.turnover, solver: run.result.solver, secs: run.result.timings?.total_s,
|
||||||
|
hash: run.result.scenario_hash?.slice(0, 8) || "–",
|
||||||
|
}));
|
||||||
|
const frontier = chartBox();
|
||||||
|
const box = card("Runs in this session", rows.length > 1
|
||||||
|
? "click a row to show that run and make it the one \u201cUse the shown result\u201d adopts; the chart is the risk/return trade-off of the runs"
|
||||||
|
: "click a row to show that run again",
|
||||||
|
el("div", {},
|
||||||
|
el("div", { class: "btnbar", style: { marginBottom: "8px" } },
|
||||||
|
button("Clear history", () => { clearRuns(); rerender(); }, "btn small")),
|
||||||
|
rows.length > 1 ? frontier : null,
|
||||||
|
dataTable([
|
||||||
|
{ key: "label", label: "Run" },
|
||||||
|
{ key: "status", label: "Status", cls: (r) => (r.status === "Optimal" ? "pos" : "muted") },
|
||||||
|
{ key: "pnl", label: "E[P&L]", num: true, fmt: (v) => fmt.money(v) },
|
||||||
|
{ key: "ret", label: "E[return]", num: true, fmt: (v) => fmt.pctSigned(v, 3) },
|
||||||
|
{ key: "cvar", label: "CVaR", num: true, fmt: (v) => fmt.money(v) },
|
||||||
|
{ key: "cvar_w", label: "CVaR of wealth", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
{ key: "n", label: "Positions", num: true, fmt: (v) => fmt.int(v) },
|
||||||
|
{ key: "turnover", label: "Turnover", num: true, fmt: (v) => fmt.pct(v, 1) },
|
||||||
|
{ key: "solver", label: "Solver" },
|
||||||
|
{ key: "secs", label: "Time", num: true, fmt: (v) => fmt.secs(v) },
|
||||||
|
{ key: "hash", label: "Scenario set", cls: "mono" },
|
||||||
|
], rows, { sortable: false, rowClass: (row) => (isSelected(row.id) ? "selected" : ""),
|
||||||
|
onRowClick: (row) => { selectRun(row.id); rerender(); } })));
|
||||||
|
if (rows.length > 1) scatterChart(frontier, {
|
||||||
|
points: rows.filter((r) => r.status !== "Infeasible").map((r) => ({ x: r.cvar_w, y: r.ret, label: r.label })),
|
||||||
|
height: 240, xLabel: "CVaR of wealth", yLabel: "expected return over the horizon",
|
||||||
|
xFormat: (v) => fmt.pct(v, 1), yFormat: (v) => fmt.pct(v, 2), labelPoints: false,
|
||||||
|
});
|
||||||
|
return box;
|
||||||
|
}
|
||||||
299
quantcore/api/ui/js/tabs/risk.js
Normal file
299
quantcore/api/ui/js/tabs/risk.js
Normal file
@ -0,0 +1,299 @@
|
|||||||
|
/* Tab 2 -- risk models: the covariance the optimiser sees and the scenarios it is given. */
|
||||||
|
|
||||||
|
import {
|
||||||
|
api, button, card, chartBox, dataTable, el, empty, field, fmt, kv, live, scenarioPayload,
|
||||||
|
segmented, select, setChildren, state, statTiles, warnBox, withBusy,
|
||||||
|
} from "../core.js";
|
||||||
|
import { barChart, heatmap, histogram } from "../charts.js";
|
||||||
|
import { info } from "../universe.js";
|
||||||
|
import { selectedRun, sharesOf } from "../runs.js";
|
||||||
|
|
||||||
|
let covResult = null, scenResult = null, noise = null;
|
||||||
|
let basis = "cap"; // which portfolio the distribution is measured for
|
||||||
|
let focus = ""; // asset shown as a single-stock distribution
|
||||||
|
let space = "simple"; // 'simple' (price returns) | 'log' (the modelling space)
|
||||||
|
|
||||||
|
export function render(node) {
|
||||||
|
node.replaceChildren();
|
||||||
|
if (!state.selection.length) { node.appendChild(empty("select candidate stocks in the Stocks tab first")); return; }
|
||||||
|
node.append(covarianceCard(), scenarioCard());
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ covariance / correlation */
|
||||||
|
function covarianceCard() {
|
||||||
|
const out = el("div", { id: "risk-cov" });
|
||||||
|
let btn;
|
||||||
|
const run = async () => {
|
||||||
|
await withBusy(btn, live("risk-cov", out), async () => {
|
||||||
|
covResult = await api.post("/model/covariance", {
|
||||||
|
symbols: state.selection, freq: state.model.freq, horizon: state.model.horizon,
|
||||||
|
covariance: scenarioPayload().covariance,
|
||||||
|
});
|
||||||
|
drawCovariance(live("risk-cov", out), covResult);
|
||||||
|
}).catch(() => {});
|
||||||
|
};
|
||||||
|
btn = button("Estimate covariance", run, "btn primary");
|
||||||
|
const body = el("div", {}, el("div", { class: "btnbar", style: { marginBottom: "10px" } }, btn,
|
||||||
|
el("span", { class: "muted", text: `${state.selection.length} stocks · ${state.model.covariance.method} · shrinkage ${state.model.covariance.shrinkage}` })), out);
|
||||||
|
if (covResult) drawCovariance(out, covResult); else out.appendChild(empty("estimate the covariance to see correlations, volatilities and conditioning"));
|
||||||
|
return card("Covariance model", "the estimator the scenario generator and the optimiser use", body);
|
||||||
|
}
|
||||||
|
|
||||||
|
function drawCovariance(out, r) {
|
||||||
|
const c = r.conditioning;
|
||||||
|
const warnings = [];
|
||||||
|
if (c.rank_deficient) warnings.push(`The estimate is singular: ${c.effective_observations} effective observations for ${c.n_assets} assets. Some portfolios look riskless to the optimiser - use shrinkage 'auto' or a longer window.`);
|
||||||
|
else if (c.under_determined) warnings.push(`Fewer effective observations (${c.effective_observations}) than assets (${c.n_assets}); shrinkage of ${fmt.pct(c.shrinkage_applied, 0)} keeps the smallest eigenvalue at ${fmt.pct(c.min_eigenvalue_ratio, 1)} of the average variance.`);
|
||||||
|
|
||||||
|
const labels = r.correlation.columns;
|
||||||
|
const vols = labels.map((s) => ({ label: s, values: [r.volatility_ann[s]] }))
|
||||||
|
.sort((a, b) => b.values[0] - a.values[0]).slice(0, 25);
|
||||||
|
const eig = r.eigenvalues.slice(0, Math.min(20, r.eigenvalues.length))
|
||||||
|
.map((v, i) => ({ label: `${i + 1}`, values: [v] }));
|
||||||
|
|
||||||
|
const hm = chartBox(), volBox = chartBox(), eigBox = chartBox();
|
||||||
|
setChildren(out,
|
||||||
|
warnBox(warnings),
|
||||||
|
statTiles([
|
||||||
|
{ label: "Assets", value: fmt.int(c.n_assets) },
|
||||||
|
{ label: "Observations", value: fmt.int(r.observations) },
|
||||||
|
{ label: "Effective obs.", value: fmt.int(c.effective_observations), delta: `${state.model.covariance.method}` },
|
||||||
|
{ label: "Shrinkage", value: fmt.pct(r.shrinkage_applied, 1), delta: "towards the diagonal" },
|
||||||
|
{ label: "Mean correlation", value: fmt.num(r.correlation_summary.mean, 2),
|
||||||
|
delta: `${fmt.num(r.correlation_summary.min, 2)} … ${fmt.num(r.correlation_summary.max, 2)}` },
|
||||||
|
{ label: "Condition number", value: fmt.compact(c.condition_number) },
|
||||||
|
{ label: "Min eigenvalue", value: fmt.pct(c.min_eigenvalue_ratio, 2), delta: "of the average variance",
|
||||||
|
deltaClass: c.rank_deficient ? "neg" : "muted" },
|
||||||
|
{ label: "Horizon", value: `${r.horizon_periods} × ${r.freq.slice(0, 1)}` },
|
||||||
|
]),
|
||||||
|
el("div", { class: "grid two", style: { marginTop: "12px" } },
|
||||||
|
card("Correlation", labels.length > 26 ? `${labels.length} stocks (labels omitted above 26)` : "of the model's returns", hm),
|
||||||
|
el("div", {},
|
||||||
|
card("Volatility p.a.", "the 25 most volatile of the selection", volBox),
|
||||||
|
card("Eigenvalue spectrum", "share of total variance per principal component", eigBox))),
|
||||||
|
);
|
||||||
|
heatmap(hm, { labels, matrix: r.correlation.data, domain: [-1, 1], valueName: "correlation" });
|
||||||
|
barChart(volBox, { items: vols, valueFormat: (v) => fmt.pct(v, 0), horizontal: true, rowHeight: 18,
|
||||||
|
labelWidth: 84, valueName: "volatility p.a." });
|
||||||
|
barChart(eigBox, { items: eig, valueFormat: (v) => fmt.pct(v, 0), horizontal: false, height: 190,
|
||||||
|
valueName: "share of variance", bottom: 26 });
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ scenarios */
|
||||||
|
function exposureFor(kind) {
|
||||||
|
const sel = state.selection;
|
||||||
|
const held = Object.entries(state.portfolio.holdings || {})
|
||||||
|
.reduce((a, [sym, n]) => a + (info(sym)?.price || 0) * Math.round(Number(n) || 0), 0);
|
||||||
|
const wealth = (Number(state.portfolio.cash) || 0) + (state.portfolio.mode === "rebalance" ? held : 0) || 1e6;
|
||||||
|
if (kind === "holdings") return { holdings: state.portfolio.holdings, value: null };
|
||||||
|
if (kind === "result") {
|
||||||
|
const r = selectedRun(); // the run shown in the Optimisation tab, not the newest
|
||||||
|
return r ? { holdings: sharesOf(r.result), value: null } : null;
|
||||||
|
}
|
||||||
|
if (kind === "equal") {
|
||||||
|
const w = Object.fromEntries(sel.map((s) => [s, 1 / sel.length]));
|
||||||
|
return { weights: w, value: wealth };
|
||||||
|
}
|
||||||
|
const caps = sel.map((s) => info(s)?.market_cap || 0);
|
||||||
|
const total = caps.reduce((a, b) => a + b, 0) || 1;
|
||||||
|
return { weights: Object.fromEntries(sel.map((s, i) => [s, caps[i] / total])), value: wealth };
|
||||||
|
}
|
||||||
|
|
||||||
|
function scenarioCard() {
|
||||||
|
const out = el("div", { id: "risk-scen" });
|
||||||
|
let btn, noiseBtn;
|
||||||
|
|
||||||
|
const run = async () => {
|
||||||
|
await withBusy(btn, live("risk-scen", out), async () => {
|
||||||
|
const exp = exposureFor(basis);
|
||||||
|
if (!exp) throw new Error("no optimisation result yet - run one in the Optimisation tab");
|
||||||
|
const body = scenarioPayload({ bins: 60, confidence: [0.95, 0.99], focus: focus ? [focus] : [] });
|
||||||
|
if (exp.holdings) Object.assign(body, { holdings: exp.holdings, cash: 0 });
|
||||||
|
else Object.assign(body, { weights: exp.weights, value: exp.value });
|
||||||
|
scenResult = await api.post("/model/scenarios", body);
|
||||||
|
drawScenarios(live("risk-scen", out), scenResult);
|
||||||
|
}).catch(() => {});
|
||||||
|
};
|
||||||
|
|
||||||
|
const noiseCheck = async () => {
|
||||||
|
await withBusy(noiseBtn, live("risk-scen", out), async () => {
|
||||||
|
const exp = exposureFor(basis);
|
||||||
|
const seeds = [0, 1, 2].map((d) => Number(state.model.seed) + d);
|
||||||
|
const results = [];
|
||||||
|
for (const seed of seeds) {
|
||||||
|
const body = scenarioPayload({ seed, bins: 20, confidence: [0.95, 0.99] });
|
||||||
|
if (exp.holdings) Object.assign(body, { holdings: exp.holdings, cash: 0 });
|
||||||
|
else Object.assign(body, { weights: exp.weights, value: exp.value });
|
||||||
|
results.push(await api.post("/model/scenarios", body));
|
||||||
|
}
|
||||||
|
noise = { seeds, results };
|
||||||
|
drawScenarios(live("risk-scen", out), scenResult, noise);
|
||||||
|
}).catch(() => {});
|
||||||
|
};
|
||||||
|
|
||||||
|
btn = button("Generate scenarios", run, "btn primary");
|
||||||
|
noiseBtn = button("Sampling-noise check (3 seeds)", noiseCheck, "btn");
|
||||||
|
|
||||||
|
const body = el("div", {},
|
||||||
|
el("div", { class: "row", style: { marginBottom: "10px" } },
|
||||||
|
field("Measured portfolio", select([
|
||||||
|
["cap", "cap-weighted selection"], ["equal", "equally weighted selection"],
|
||||||
|
["holdings", "current holdings"], ["result", "the optimisation result on screen"]], basis, (v) => { basis = v; run(); })),
|
||||||
|
field("Single-stock view", select([["", "none"], ...state.selection.map((s) => [s, s])], focus, (v) => { focus = v; run(); })),
|
||||||
|
el("div", { class: "field auto" }, el("label", { text: " " }), el("div", { class: "btnbar" }, btn, noiseBtn))),
|
||||||
|
out);
|
||||||
|
if (scenResult) drawScenarios(out, scenResult, noise);
|
||||||
|
else out.appendChild(empty("generate the scenario set to see the horizon distribution of the portfolio and of each stock"));
|
||||||
|
return card("Scenarios", `${state.model.scenario_model === "gaussian_exact" ? "GBM (the exact Gaussian mode draws no scenarios)" : state.model.scenario_model} · ${state.model.sampling} sampling · ${state.model.horizon} horizon`, body);
|
||||||
|
}
|
||||||
|
|
||||||
|
function drawScenarios(out, r, noiseData) {
|
||||||
|
if (!r) return;
|
||||||
|
const d = r.description, p = r.portfolio;
|
||||||
|
const tiles = [
|
||||||
|
{ label: "Scenarios", value: fmt.int(d.n_scenarios), delta: d.weighted ? "probability weighted" : "equiprobable" },
|
||||||
|
{ label: "Effective sample", value: fmt.int(d.effective_sample_size),
|
||||||
|
delta: d.weighted ? `${fmt.pct(d.effective_sample_size / d.n_scenarios, 0)} of the draws` : "no tilt" },
|
||||||
|
{ label: "Scenario set", value: d.hash?.slice(0, 8) || "–", delta: "same inputs ⇒ same set", title: d.hash },
|
||||||
|
{ label: "Shocks", value: d.shock_hash?.slice(0, 8) || "–", delta: "drift removed" },
|
||||||
|
];
|
||||||
|
if (p) {
|
||||||
|
tiles.push(
|
||||||
|
{ label: "Expected P&L", value: fmt.money(p.expected_pnl), delta: fmt.pctSigned(p.expected_pnl / p.value, 2),
|
||||||
|
deltaClass: p.expected_pnl >= 0 ? "pos" : "neg" },
|
||||||
|
{ label: "P&L volatility", value: fmt.money(p.volatility) },
|
||||||
|
{ label: "VaR 95 %", value: fmt.money(p.risk["0.95"].var) },
|
||||||
|
{ label: "CVaR 95 %", value: fmt.money(p.risk["0.95"].cvar), delta: fmt.pct(p.risk["0.95"].cvar / p.value, 2) },
|
||||||
|
{ label: "CVaR 99 %", value: fmt.money(p.risk["0.99"].cvar), delta: fmt.pct(p.risk["0.99"].cvar / p.value, 2) },
|
||||||
|
{ label: "Skewness", value: fmt.num(p.skewness, 2), delta: `kurtosis ${fmt.num(p.kurtosis, 1)}` });
|
||||||
|
}
|
||||||
|
|
||||||
|
const hist = chartBox(280), assetBox = chartBox(240);
|
||||||
|
const parts = [statTiles(tiles)];
|
||||||
|
|
||||||
|
if (p) {
|
||||||
|
parts.push(el("div", { class: "grid two", style: { marginTop: "12px" } },
|
||||||
|
card(`P&L over one ${state.model.horizon}`, `${basisLabel()} · ${fmt.compact(p.value)} CHF`, hist,
|
||||||
|
el("div", { class: "note", style: { marginTop: "6px" } },
|
||||||
|
`Losses beyond the 95 % VaR are shaded; CVaR is their probability-weighted average.`)),
|
||||||
|
card("Quantiles of the P&L", "probability-weighted, in CHF", quantileTable(p))));
|
||||||
|
}
|
||||||
|
if (r.asset_histograms && Object.keys(r.asset_histograms).length) {
|
||||||
|
const sym = Object.keys(r.asset_histograms)[0];
|
||||||
|
parts.push(card(`${sym} — simple return over one ${state.model.horizon}`, info(sym)?.name || "", assetBox));
|
||||||
|
}
|
||||||
|
if (noiseData) parts.push(noiseCard(noiseData));
|
||||||
|
parts.push(summaryCard(r));
|
||||||
|
parts.push(card("Provenance", "the inputs this scenario set depends on - any run with the same key reuses it",
|
||||||
|
kv(Object.entries(r.scenario_key || {}).map(([k, v]) => [
|
||||||
|
k, Array.isArray(v) ? `${v.length}: ${v.slice(0, 8).join(", ")}${v.length > 8 ? " …" : ""}`
|
||||||
|
: typeof v === "object" && v !== null ? Object.entries(v).map(([a, b]) => `${a} ${b}`).join(", ")
|
||||||
|
: String(v)]))));
|
||||||
|
|
||||||
|
setChildren(out, ...parts);
|
||||||
|
|
||||||
|
if (p) {
|
||||||
|
histogram(hist, {
|
||||||
|
edges: p.histogram.edges, mass: p.histogram.mass, height: 280,
|
||||||
|
valueFormat: (v) => fmt.compact(v), tailFrom: -p.risk["0.95"].var,
|
||||||
|
mainLabel: "P&L outcomes", tailLabel: "worst 5 %",
|
||||||
|
markers: [
|
||||||
|
{ value: -p.risk["0.95"].var, label: `VaR 95 % ${fmt.compact(p.risk["0.95"].var)}` },
|
||||||
|
{ value: -p.risk["0.95"].cvar, label: `CVaR 95 % ${fmt.compact(p.risk["0.95"].cvar)}` },
|
||||||
|
{ value: p.expected_pnl, label: `mean ${fmt.compact(p.expected_pnl)}` },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if (r.asset_histograms) {
|
||||||
|
const sym = Object.keys(r.asset_histograms)[0];
|
||||||
|
if (sym) {
|
||||||
|
const h = r.asset_histograms[sym];
|
||||||
|
histogram(assetBox, { edges: h.edges, mass: h.mass, height: 240, valueFormat: (v) => fmt.pct(v, 0),
|
||||||
|
mainLabel: `${sym} outcomes` });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function basisLabel() {
|
||||||
|
return { cap: "cap-weighted selection", equal: "equally weighted selection", holdings: "current holdings",
|
||||||
|
result: "the optimisation result on screen" }[basis];
|
||||||
|
}
|
||||||
|
|
||||||
|
function quantileTable(p) {
|
||||||
|
const rows = Object.entries(p.quantiles).map(([q, v]) => ({ q: `${(Number(q) * 100).toFixed(0)} %`, v, r: v / p.value }));
|
||||||
|
rows.push({ q: "mean", v: p.expected_pnl, r: p.expected_pnl / p.value });
|
||||||
|
rows.push({ q: "worst", v: p.min, r: p.min / p.value });
|
||||||
|
rows.push({ q: "best", v: p.max, r: p.max / p.value });
|
||||||
|
return dataTable([
|
||||||
|
{ key: "q", label: "Quantile" },
|
||||||
|
{ key: "v", label: "P&L (CHF)", num: true, fmt: (v) => fmt.money(v), cls: (r) => (r.v >= 0 ? "pos" : "neg") },
|
||||||
|
{ key: "r", label: "of wealth", num: true, fmt: (v) => fmt.pctSigned(v, 2) },
|
||||||
|
], rows, { sortable: false });
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Per-stock scenario table. Price returns by default: a log-return of -2.0 over a quarter is a
|
||||||
|
price return of -86 %, and only the latter is what a position can actually lose. */
|
||||||
|
function summaryCard(r) {
|
||||||
|
const slot = el("div");
|
||||||
|
const fill = () => setChildren(slot, summaryTable(r));
|
||||||
|
fill();
|
||||||
|
const head = el("div", { class: "row", style: { marginBottom: "8px" } },
|
||||||
|
field("Shown as", segmented([["simple", "price returns"], ["log", "log-returns"]], space,
|
||||||
|
(v) => { space = v; fill(); })));
|
||||||
|
return card("Scenario set per stock",
|
||||||
|
space === "simple"
|
||||||
|
? `probability-weighted moments and quantiles of the horizon price return (e^x − 1, never below −100 %)`
|
||||||
|
: `probability-weighted moments and quantiles of the horizon log-return (the space the model works in)`,
|
||||||
|
head, slot);
|
||||||
|
}
|
||||||
|
|
||||||
|
function summaryTable(r) {
|
||||||
|
const rows = r.summary.map((s) => ({ ...s, mu: r.expected_returns_ann?.[s.symbol] }));
|
||||||
|
const k = (name) => (space === "simple" ? `${name}_simple` : name);
|
||||||
|
const cols = [
|
||||||
|
{ key: "symbol", label: "Symbol", cls: "sym" },
|
||||||
|
{ key: "mu", label: "E[r] p.a.", num: true, fmt: (v) => fmt.pct(v, 1), title: "the expected-return model" },
|
||||||
|
];
|
||||||
|
if (space === "simple") {
|
||||||
|
cols.push({ key: "expected_simple", label: "E[r] horizon", num: true, fmt: (v) => fmt.pct(v, 2),
|
||||||
|
title: "the model's expected price return over the horizon" });
|
||||||
|
}
|
||||||
|
cols.push(
|
||||||
|
{ key: k("mean"), label: space === "simple" ? "scenario mean" : "mean log", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
{ key: k("std"), label: "sd", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
{ key: k("q05"), label: "5 %", num: true, fmt: (v) => fmt.pct(v, 1) },
|
||||||
|
{ key: k("q50"), label: "median", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
{ key: k("q95"), label: "95 %", num: true, fmt: (v) => fmt.pct(v, 1) },
|
||||||
|
{ key: k("min"), label: "worst", num: true, fmt: (v) => fmt.pct(v, 1), cls: "neg" },
|
||||||
|
{ key: k("max"), label: "best", num: true, fmt: (v) => fmt.pct(v, 1), cls: "pos" },
|
||||||
|
);
|
||||||
|
return dataTable(cols, rows, { sortKey: k("std"), sortDir: -1 });
|
||||||
|
}
|
||||||
|
|
||||||
|
function noiseCard(nd) {
|
||||||
|
const rows = nd.results.map((r, i) => ({
|
||||||
|
seed: nd.seeds[i], pnl: r.portfolio.expected_pnl, vol: r.portfolio.volatility,
|
||||||
|
cvar95: r.portfolio.risk["0.95"].cvar, cvar99: r.portfolio.risk["0.99"].cvar, hash: r.description.hash.slice(0, 8),
|
||||||
|
}));
|
||||||
|
const spread = (key) => {
|
||||||
|
const v = rows.map((r) => r[key]);
|
||||||
|
const mean = v.reduce((a, b) => a + b, 0) / v.length;
|
||||||
|
const sd = Math.sqrt(v.reduce((a, b) => a + (b - mean) ** 2, 0) / (v.length - 1));
|
||||||
|
return { mean, sd, rel: sd / Math.abs(mean) };
|
||||||
|
};
|
||||||
|
const c95 = spread("cvar95"), c99 = spread("cvar99");
|
||||||
|
return card("Sampling noise across seeds", "the same model and portfolio, three independent scenario sets",
|
||||||
|
el("div", {},
|
||||||
|
statTiles([
|
||||||
|
{ label: "CVaR 95 % spread", value: fmt.pct(c95.rel, 2), delta: `sd ${fmt.money(c95.sd)} of ${fmt.money(c95.mean)}` },
|
||||||
|
{ label: "CVaR 99 % spread", value: fmt.pct(c99.rel, 2), delta: `sd ${fmt.money(c99.sd)} of ${fmt.money(c99.mean)}` },
|
||||||
|
]),
|
||||||
|
el("div", { style: { marginTop: "10px" } }, dataTable([
|
||||||
|
{ key: "seed", label: "Seed", num: true, fmt: (v) => fmt.int(v) },
|
||||||
|
{ key: "hash", label: "Scenario set", cls: "mono" },
|
||||||
|
{ key: "pnl", label: "E[P&L]", num: true, fmt: (v) => fmt.money(v) },
|
||||||
|
{ key: "vol", label: "sd of P&L", num: true, fmt: (v) => fmt.money(v) },
|
||||||
|
{ key: "cvar95", label: "CVaR 95 %", num: true, fmt: (v) => fmt.money(v) },
|
||||||
|
{ key: "cvar99", label: "CVaR 99 %", num: true, fmt: (v) => fmt.money(v) },
|
||||||
|
], rows, { sortable: false }))));
|
||||||
|
}
|
||||||
217
quantcore/api/ui/js/tabs/stocks.js
Normal file
217
quantcore/api/ui/js/tabs/stocks.js
Normal file
@ -0,0 +1,217 @@
|
|||||||
|
/* Tab 1 -- stock data: pick the candidate stocks and look at what the models are fed. */
|
||||||
|
|
||||||
|
import {
|
||||||
|
api, button, card, changed, chartBox, dataTable, el, empty, errorBox, field, fmt, input,
|
||||||
|
live, parseDate, segmented, select, setChildren, state, withBusy,
|
||||||
|
} from "../core.js";
|
||||||
|
import { barChart, lineChart, scatterChart } from "../charts.js";
|
||||||
|
import { filterRows, info, universe } from "../universe.js";
|
||||||
|
|
||||||
|
const RANGES = [["1y", 1], ["3y", 3], ["5y", 5], ["10y", 10], ["max", 0]];
|
||||||
|
let range = "5y";
|
||||||
|
let analysis = null; // {prices, risk, perf} of the last fetch
|
||||||
|
|
||||||
|
export async function render(node) {
|
||||||
|
node.replaceChildren(el("div", { class: "empty" }, el("span", { class: "spinner" }), document.createTextNode(" loading the universe …")));
|
||||||
|
let uni;
|
||||||
|
try { uni = await universe(); } catch (err) { node.replaceChildren(errorBox(err)); return; }
|
||||||
|
node.replaceChildren();
|
||||||
|
node.append(pickerCard(uni), analysisCard());
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ stock picker */
|
||||||
|
function pickerCard(uni) {
|
||||||
|
const u = state.universe;
|
||||||
|
const body = el("div");
|
||||||
|
const tableSlot = el("div");
|
||||||
|
const summary = el("div", { class: "row", style: { marginBottom: "8px" } });
|
||||||
|
|
||||||
|
const draw = () => {
|
||||||
|
const rows = filterRows(uni.rows, { region: u.region, sector: u.sector, country: u.country, search: u.search });
|
||||||
|
const selected = new Set(state.selection);
|
||||||
|
summary.replaceChildren(
|
||||||
|
el("span", { class: "muted", text: `${rows.length} of ${uni.rows.length} stocks match · ${selected.size} selected` }),
|
||||||
|
el("span", { class: "spacer", style: { marginLeft: "auto" } }),
|
||||||
|
el("div", { class: "btnbar" },
|
||||||
|
button(`Select top ${u.top}`, () => {
|
||||||
|
state.selection = rows.slice(0, Number(u.top)).map((r) => r.symbol);
|
||||||
|
changed("selection", "stocks"); draw();
|
||||||
|
}, "btn small primary"),
|
||||||
|
button("Add all matching", () => {
|
||||||
|
const add = rows.slice(0, 500).map((r) => r.symbol);
|
||||||
|
state.selection = [...new Set([...state.selection, ...add])];
|
||||||
|
changed("selection", "stocks"); draw();
|
||||||
|
}, "btn small"),
|
||||||
|
button("Clear", () => { state.selection = []; changed("selection", "stocks"); draw(); }, "btn small")));
|
||||||
|
|
||||||
|
tableSlot.replaceChildren(dataTable([
|
||||||
|
{ key: "selected", label: "", render: (r) => {
|
||||||
|
const box = el("input", { type: "checkbox" });
|
||||||
|
box.checked = selected.has(r.symbol);
|
||||||
|
box.addEventListener("change", () => {
|
||||||
|
state.selection = box.checked
|
||||||
|
? [...new Set([...state.selection, r.symbol])]
|
||||||
|
: state.selection.filter((s) => s !== r.symbol);
|
||||||
|
changed("selection", "stocks");
|
||||||
|
draw();
|
||||||
|
});
|
||||||
|
return box;
|
||||||
|
} },
|
||||||
|
{ key: "symbol", label: "Symbol", cls: "sym" },
|
||||||
|
{ key: "name", label: "Name", cls: "name" },
|
||||||
|
{ key: "sector", label: "Sector" },
|
||||||
|
{ key: "country", label: "Country" },
|
||||||
|
{ key: "market_cap", label: "Market cap (CHF)", num: true, fmt: (v) => fmt.compact(v) },
|
||||||
|
{ key: "market_weight", label: "Weight", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
{ key: "price", label: "Price (CHF)", num: true, fmt: (v) => fmt.money2(v) },
|
||||||
|
{ key: "price_coverage", label: "Coverage", num: true, fmt: (v) => fmt.pct(v, 1) },
|
||||||
|
], rows, { sortKey: "market_cap", sortDir: -1, limit: 400 }));
|
||||||
|
};
|
||||||
|
|
||||||
|
const sectors = ["", ...Object.keys(uni.facets.sectors)];
|
||||||
|
const countries = ["", ...Object.keys(uni.facets.countries).sort()];
|
||||||
|
body.append(
|
||||||
|
el("div", { class: "row" },
|
||||||
|
field("Region", select([["", "world"], ...Object.keys(uni.facets.regions).map((r) => [r, r])], u.region,
|
||||||
|
(v) => { u.region = v; changed("universe", "stocks"); draw(); })),
|
||||||
|
field("Sector", select(sectors.map((s) => [s, s || "all sectors"]), u.sector,
|
||||||
|
(v) => { u.sector = v; changed("universe", "stocks"); draw(); })),
|
||||||
|
field("Country", select(countries.map((s) => [s, s || "all countries"]), u.country || "",
|
||||||
|
(v) => { u.country = v; changed("universe", "stocks"); draw(); })),
|
||||||
|
field("Search", input("search", u.search, (v) => { u.search = v; changed("universe", "stocks"); draw(); },
|
||||||
|
{ placeholder: "symbol or name" }))),
|
||||||
|
summary, tableSlot);
|
||||||
|
draw();
|
||||||
|
return card("Investment universe", `${uni.n_universe} stocks, market caps as of ${fmt.date(uni.as_of)}`, body);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ time series and statistics */
|
||||||
|
function analysisCard() {
|
||||||
|
const body = el("div");
|
||||||
|
const out = el("div", { id: "stocks-analysis" });
|
||||||
|
const controls = el("div", { class: "row", style: { marginBottom: "10px" } });
|
||||||
|
let btn;
|
||||||
|
|
||||||
|
const run = async () => {
|
||||||
|
const symbols = state.selection;
|
||||||
|
const panel = live("stocks-analysis", out);
|
||||||
|
if (!symbols.length) { panel.replaceChildren(empty("select stocks above to see prices, risk and performance")); return; }
|
||||||
|
await withBusy(btn, panel, async () => {
|
||||||
|
const years = RANGES.find(([k]) => k === range)?.[1] ?? 5;
|
||||||
|
const start = years ? new Date(Date.now() - years * 365.25 * 864e5).toISOString().slice(0, 10) : null;
|
||||||
|
const list = symbols.slice(0, 150).join(",");
|
||||||
|
const [prices, risk, perf] = await Promise.all([
|
||||||
|
api.get("/prices", { symbols: list, start, freq: state.data.freq, currency: "base" }),
|
||||||
|
api.post("/risk/assets", { symbols: symbols.slice(0, 150), freq: state.data.freq, horizon: state.model.horizon,
|
||||||
|
start, currency: "base", space: "value", drift: "zero" }),
|
||||||
|
api.post("/performance/assets", { symbols: symbols.slice(0, 150), freq: state.data.freq, start, currency: "base" }),
|
||||||
|
]);
|
||||||
|
analysis = { prices, risk, perf };
|
||||||
|
drawAnalysis(live("stocks-analysis", panel), prices, risk, perf);
|
||||||
|
}).catch(() => {});
|
||||||
|
};
|
||||||
|
|
||||||
|
btn = button("Load data", run, "btn primary");
|
||||||
|
controls.append(
|
||||||
|
field("History", segmented(RANGES.map(([k]) => [k, k]), range, (v) => { range = v; run(); })),
|
||||||
|
field("Frequency", select([["daily", "daily"], ["weekly", "weekly"], ["monthly", "monthly"]], state.data.freq,
|
||||||
|
(v) => { state.data.freq = v; changed("data", "stocks"); run(); })),
|
||||||
|
el("div", { class: "field auto" }, el("label", { text: " " }), btn));
|
||||||
|
body.append(controls, out);
|
||||||
|
out.appendChild(empty("select stocks above, then load the data"));
|
||||||
|
if (state.selection.length && analysis) drawAnalysis(out, analysis.prices, analysis.risk, analysis.perf);
|
||||||
|
return card("Time series, risk and performance", "prices in CHF; log-returns; the window is the range above", body);
|
||||||
|
}
|
||||||
|
|
||||||
|
function drawAnalysis(out, prices, risk, perf) {
|
||||||
|
const { index, columns, data } = prices;
|
||||||
|
const x = index.map(parseDate);
|
||||||
|
const shown = [...columns].sort((a, b) => (info(b)?.market_cap ?? 0) - (info(a)?.market_cap ?? 0)).slice(0, 8);
|
||||||
|
|
||||||
|
// indexed prices (a common base is the only honest way to put several stocks on one axis)
|
||||||
|
const series = shown.map((c) => {
|
||||||
|
const j = columns.indexOf(c);
|
||||||
|
const col = data.map((row) => row[j]);
|
||||||
|
const base = col.find((v) => Number.isFinite(v)) ?? 1;
|
||||||
|
return { name: c, points: col.map((v, i) => [x[i], Number.isFinite(v) ? (v / base) * 100 : NaN]) };
|
||||||
|
});
|
||||||
|
const dd = shown.map((c) => {
|
||||||
|
const j = columns.indexOf(c);
|
||||||
|
let peak = -Infinity;
|
||||||
|
const pts = [];
|
||||||
|
data.forEach((row, i) => {
|
||||||
|
const v = row[j];
|
||||||
|
if (!Number.isFinite(v)) return;
|
||||||
|
peak = Math.max(peak, v);
|
||||||
|
pts.push([x[i], v / peak - 1]);
|
||||||
|
});
|
||||||
|
return { name: c, points: pts };
|
||||||
|
});
|
||||||
|
|
||||||
|
const perfBy = Object.fromEntries(perf.map((p) => [p.symbol, p]));
|
||||||
|
const riskBy = Object.fromEntries(risk.map((p) => [p.symbol, p]));
|
||||||
|
const caps = state.selection.map((s) => ({ label: s, values: [info(s)?.market_cap ?? NaN] }))
|
||||||
|
.filter((r) => Number.isFinite(r.values[0])).sort((a, b) => b.values[0] - a.values[0]).slice(0, 20);
|
||||||
|
|
||||||
|
const priceChart = chartBox(260), ddChart = chartBox(220), capChart = chartBox(), scatter = chartBox(280);
|
||||||
|
setChildren(out,
|
||||||
|
el("div", { class: "grid two" },
|
||||||
|
card("Price, indexed to 100", `${shown.length} of ${columns.length} stocks · ${index.length} ${state.data.freq} observations`, priceChart),
|
||||||
|
card("Drawdown from the running peak", "same stocks", ddChart)),
|
||||||
|
el("div", { class: "grid two" },
|
||||||
|
card("Risk and return", `annualised, from ${fmt.date(index[0])}`, scatter),
|
||||||
|
card("Market capitalisation", "the 20 largest of the selection, CHF", capChart)),
|
||||||
|
card("Per-stock statistics",
|
||||||
|
`risk over a ${state.model.horizon} horizon, ${state.data.freq} data; VaR and CVaR as a share of the `
|
||||||
|
+ `position's value, volatility only (no drift extrapolated into the horizon)`,
|
||||||
|
statsTable(state.selection, riskBy, perfBy)),
|
||||||
|
);
|
||||||
|
|
||||||
|
lineChart(priceChart, { series, height: 260, xFormat: (v) => new Date(v).toISOString().slice(0, 7),
|
||||||
|
yFormat: (v) => fmt.num(v, 0) });
|
||||||
|
lineChart(ddChart, { series: dd, height: 220, xFormat: (v) => new Date(v).toISOString().slice(0, 7),
|
||||||
|
yFormat: (v) => fmt.pct(v, 0), area: dd.length === 1 });
|
||||||
|
barChart(capChart, { items: caps, valueFormat: (v) => fmt.compact(v), horizontal: true, rowHeight: 20,
|
||||||
|
labelWidth: 84, valueName: "market cap" });
|
||||||
|
scatterChart(scatter, {
|
||||||
|
points: state.selection.filter((s) => perfBy[s]).map((s) => ({
|
||||||
|
x: perfBy[s].annualised_volatility, y: perfBy[s].annualised_return, label: s })),
|
||||||
|
height: 280, xLabel: "volatility p.a.", yLabel: "return p.a.",
|
||||||
|
xFormat: (v) => fmt.pct(v, 0), yFormat: (v) => fmt.pct(v, 0), labelPoints: state.selection.length <= 14,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function statsTable(symbols, riskBy, perfBy) {
|
||||||
|
const rows = symbols.filter((s) => riskBy[s] || perfBy[s]).map((s) => ({
|
||||||
|
symbol: s,
|
||||||
|
vol: perfBy[s]?.annualised_volatility,
|
||||||
|
vol_ewm: riskBy[s]?.vol_ewm_ann,
|
||||||
|
ret: perfBy[s]?.annualised_return,
|
||||||
|
sharpe: perfBy[s]?.sharpe,
|
||||||
|
mdd: perfBy[s]?.max_drawdown,
|
||||||
|
skew: perfBy[s]?.skewness,
|
||||||
|
kurt: perfBy[s]?.excess_kurtosis,
|
||||||
|
var95: riskBy[s]?.var95_normal_ewm,
|
||||||
|
cvar95: riskBy[s]?.cvar95_normal_ewm,
|
||||||
|
cvar95_hist: riskBy[s]?.cvar95_hist,
|
||||||
|
cvar99_hist: riskBy[s]?.cvar99_hist,
|
||||||
|
n: perfBy[s]?.n_obs,
|
||||||
|
}));
|
||||||
|
return dataTable([
|
||||||
|
{ key: "symbol", label: "Symbol", cls: "sym" },
|
||||||
|
{ key: "ret", label: "Return p.a.", num: true, fmt: (v) => fmt.pct(v, 1), cls: (r) => (r.ret >= 0 ? "pos" : "neg") },
|
||||||
|
{ key: "vol", label: "Vol p.a.", num: true, fmt: (v) => fmt.pct(v, 1) },
|
||||||
|
{ key: "vol_ewm", label: "Vol p.a. (EWM)", num: true, fmt: (v) => fmt.pct(v, 1),
|
||||||
|
title: "exponentially weighted: the volatility the models use today" },
|
||||||
|
{ key: "sharpe", label: "Sharpe", num: true, fmt: (v) => fmt.num(v, 2) },
|
||||||
|
{ key: "mdd", label: "Max drawdown", num: true, fmt: (v) => fmt.pct(v, 0), cls: "neg" },
|
||||||
|
{ key: "skew", label: "Skew", num: true, fmt: (v) => fmt.num(v, 2) },
|
||||||
|
{ key: "kurt", label: "Excess kurtosis", num: true, fmt: (v) => fmt.num(v, 1) },
|
||||||
|
{ key: "var95", label: "VaR 95 %", num: true, fmt: (v) => fmt.pct(v, 2),
|
||||||
|
title: "normal, EWM volatility, over the horizon; the loss as a share of the position's value" },
|
||||||
|
{ key: "cvar95", label: "CVaR 95 %", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
{ key: "cvar95_hist", label: "CVaR 95 % hist.", num: true, fmt: (v) => fmt.pct(v, 2), title: "empirical, same horizon" },
|
||||||
|
{ key: "cvar99_hist", label: "CVaR 99 % hist.", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
{ key: "n", label: "Obs.", num: true, fmt: (v) => fmt.int(v) },
|
||||||
|
], rows, { sortKey: "vol", sortDir: -1 });
|
||||||
|
}
|
||||||
172
quantcore/api/ui/js/tabs/views.js
Normal file
172
quantcore/api/ui/js/tabs/views.js
Normal file
@ -0,0 +1,172 @@
|
|||||||
|
/* Tab 3 -- expected returns: the implied (equilibrium) returns, the views and the posterior. */
|
||||||
|
|
||||||
|
import {
|
||||||
|
api, button, card, changed, chartBox, dataTable, el, empty, field, fmt, input, live, select, setChildren,
|
||||||
|
state, statTiles, viewPayload, withBusy,
|
||||||
|
} from "../core.js";
|
||||||
|
import { barChart } from "../charts.js";
|
||||||
|
import { info } from "../universe.js";
|
||||||
|
|
||||||
|
let result = null;
|
||||||
|
|
||||||
|
export function render(node) {
|
||||||
|
node.replaceChildren();
|
||||||
|
if (!state.selection.length) { node.appendChild(empty("select candidate stocks in the Stocks tab first")); return; }
|
||||||
|
node.append(viewsCard(), resultCard());
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ the views */
|
||||||
|
function viewsCard() {
|
||||||
|
const body = el("div");
|
||||||
|
const list = el("div");
|
||||||
|
|
||||||
|
const drawList = () => {
|
||||||
|
list.replaceChildren();
|
||||||
|
if (!state.views.length) { list.appendChild(el("div", { class: "note", text: "No views: the posterior equals the market-implied (equilibrium) returns." })); return; }
|
||||||
|
list.appendChild(dataTable([
|
||||||
|
{ key: "label", label: "View", render: (v) => document.createTextNode(describe(v)) },
|
||||||
|
{ key: "confidence", label: "Confidence", num: true, fmt: (v) => fmt.pct(v, 0) },
|
||||||
|
{ key: "del", label: "", render: (v) => button("remove", () => {
|
||||||
|
state.views = state.views.filter((x) => x !== v); changed("views", "views"); drawList(); }, "btn small") },
|
||||||
|
], state.views, { sortable: false }));
|
||||||
|
};
|
||||||
|
drawList();
|
||||||
|
|
||||||
|
const draft = { kind: "sentiment", symbol: state.selection[0] || "", sentiment: 0.5, confidence: 0.5, strength: 0.25,
|
||||||
|
assets: "", q: 0.1 };
|
||||||
|
const form = el("div");
|
||||||
|
const drawForm = () => {
|
||||||
|
form.replaceChildren(
|
||||||
|
el("div", { class: "row" },
|
||||||
|
field("Kind", select([["sentiment", "sentiment on one stock"], ["explicit", "explicit return view"]], draft.kind,
|
||||||
|
(v) => { draft.kind = v; drawForm(); })),
|
||||||
|
draft.kind === "sentiment"
|
||||||
|
? field("Stock", select(state.selection.map((s) => [s, `${s} — ${info(s)?.name || ""}`.slice(0, 44)]), draft.symbol,
|
||||||
|
(v) => { draft.symbol = v; }))
|
||||||
|
: field("Portfolio of the view", input("text", draft.assets, (v) => { draft.assets = v; },
|
||||||
|
{ placeholder: "AAPL:1, MSFT:-1" }), "weights of a long/short view portfolio"),
|
||||||
|
draft.kind === "sentiment"
|
||||||
|
? field("Sentiment −1 … +1", input("number", draft.sentiment, (v) => { draft.sentiment = v; },
|
||||||
|
{ min: -1, max: 1, step: 0.1 }))
|
||||||
|
: field("Expected return p.a.", input("number", draft.q, (v) => { draft.q = v; }, { step: 0.01 }),
|
||||||
|
"as a fraction, e.g. 0.12"),
|
||||||
|
field("Confidence", input("number", draft.confidence, (v) => { draft.confidence = v; },
|
||||||
|
{ min: 0.01, max: 1, step: 0.05 }), "Idzorek: 1 = fully trust the view"),
|
||||||
|
draft.kind === "sentiment"
|
||||||
|
? field("Strength", input("number", draft.strength, (v) => { draft.strength = v; }, { min: 0, step: 0.05 }),
|
||||||
|
"return added at sentiment +1")
|
||||||
|
: null,
|
||||||
|
el("div", { class: "field auto" }, el("label", { text: " " }), button("Add view", () => {
|
||||||
|
const v = draft.kind === "sentiment"
|
||||||
|
? { kind: "sentiment", symbol: draft.symbol, sentiment: Number(draft.sentiment),
|
||||||
|
confidence: Number(draft.confidence), strength: Number(draft.strength) }
|
||||||
|
: { kind: "explicit", assets: parseAssets(draft.assets), q: Number(draft.q), confidence: Number(draft.confidence) };
|
||||||
|
if (v.kind === "explicit" && !Object.keys(v.assets).length) { alert("give the view portfolio as SYMBOL:weight pairs"); return; }
|
||||||
|
state.views = [...state.views, v];
|
||||||
|
changed("views", "views"); drawList();
|
||||||
|
}, "btn primary"))));
|
||||||
|
};
|
||||||
|
drawForm();
|
||||||
|
|
||||||
|
body.append(list, el("div", { style: { marginTop: "10px" } }, form));
|
||||||
|
return card("Views", "Black-Litterman: a view moves the posterior away from the market-implied return, by as much as its confidence allows", body);
|
||||||
|
}
|
||||||
|
|
||||||
|
function describe(v) {
|
||||||
|
if (v.kind === "sentiment") return `${v.symbol}: sentiment ${v.sentiment > 0 ? "+" : ""}${v.sentiment} (strength ${fmt.pct(v.strength, 0)})`;
|
||||||
|
const parts = Object.entries(v.assets).map(([s, w]) => `${w > 0 ? "+" : ""}${w} ${s}`).join(" ");
|
||||||
|
return `${parts} → ${fmt.pct(v.q, 1)} p.a.`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseAssets(text) {
|
||||||
|
const out = {};
|
||||||
|
for (const part of String(text).split(/[,;]/)) {
|
||||||
|
const [s, w] = part.split(":").map((x) => x.trim());
|
||||||
|
if (s && w !== undefined && !Number.isNaN(Number(w))) out[s.toUpperCase()] = Number(w);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ the model output */
|
||||||
|
function resultCard() {
|
||||||
|
const out = el("div", { id: "bl-result" });
|
||||||
|
let btn;
|
||||||
|
const run = async () => {
|
||||||
|
await withBusy(btn, live("bl-result", out), async () => {
|
||||||
|
result = await api.post("/model/expected-returns", {
|
||||||
|
symbols: state.selection, max_assets: state.selection.length, views: viewPayload(),
|
||||||
|
tau: Number(state.model.bl_tau), years: Number(state.model.bl_years),
|
||||||
|
omega_method: state.model.bl_omega_method || "idzorek",
|
||||||
|
});
|
||||||
|
draw(live("bl-result", out), result);
|
||||||
|
}).catch(() => {});
|
||||||
|
};
|
||||||
|
btn = button("Compute expected returns", run, "btn primary");
|
||||||
|
const body = el("div", {},
|
||||||
|
el("div", { class: "row", style: { marginBottom: "10px" } },
|
||||||
|
field("Ω (view uncertainty)", select([["idzorek", "Idzorek (from the confidence)"], ["he_litterman", "He-Litterman"]],
|
||||||
|
state.model.bl_omega_method || "idzorek", (v) => { state.model.bl_omega_method = v; changed("model", "views"); run(); })),
|
||||||
|
field("τ", input("number", state.model.bl_tau, (v) => { state.model.bl_tau = v; changed("model", "views"); }, { step: 0.01, min: 0.001 })),
|
||||||
|
field("History (years)", input("number", state.model.bl_years, (v) => { state.model.bl_years = v; changed("model", "views"); }, { min: 1, max: 20 })),
|
||||||
|
el("div", { class: "field auto" }, el("label", { text: " " }), btn)),
|
||||||
|
out);
|
||||||
|
if (result) draw(out, result); else out.appendChild(empty("compute the model to see implied and posterior expected returns"));
|
||||||
|
return card("Implied and posterior expected returns", "reverse optimisation of the cap-weighted market, then the views", body);
|
||||||
|
}
|
||||||
|
|
||||||
|
function draw(out, r) {
|
||||||
|
const symbols = Object.keys(r.prior_ann);
|
||||||
|
const rows = symbols.map((s) => ({
|
||||||
|
symbol: s, name: info(s)?.name || s, weight: r.market_weights[s], vol: r.volatility_ann[s],
|
||||||
|
prior: r.prior_ann[s], post: r.posterior_ann[s], change: r.posterior_ann[s] - r.prior_ann[s],
|
||||||
|
sharpe: r.volatility_ann[s] ? (r.posterior_ann[s] - r.risk_free_ann) / r.volatility_ann[s] : null,
|
||||||
|
}));
|
||||||
|
const moved = [...rows].sort((a, b) => Math.abs(b.change) - Math.abs(a.change));
|
||||||
|
const top = moved.slice(0, 20);
|
||||||
|
const chart = chartBox(), changeChart = chartBox();
|
||||||
|
|
||||||
|
setChildren(out,
|
||||||
|
statTiles([
|
||||||
|
{ label: "Market return p.a.", value: fmt.pct(r.market_return_ann, 2), delta: "arithmetic, cap-weighted index" },
|
||||||
|
{ label: "Market volatility", value: fmt.pct(r.market_volatility_ann, 1) },
|
||||||
|
{ label: "Implied risk aversion δ", value: fmt.num(r.delta, 2) },
|
||||||
|
{ label: "τ", value: fmt.num(r.tau, 3) },
|
||||||
|
{ label: "Risk-free", value: fmt.pct(r.risk_free_ann, 2) },
|
||||||
|
{ label: "Views", value: fmt.int(r.views.length), delta: r.omega_method },
|
||||||
|
]),
|
||||||
|
el("div", { class: "grid two", style: { marginTop: "12px" } },
|
||||||
|
card("Implied vs posterior", `the ${top.length} stocks the views move most`, chart),
|
||||||
|
card("Change from the views", "posterior − implied, p.a.", changeChart)),
|
||||||
|
r.views.length
|
||||||
|
? card("How the views were translated", "q is the view's expected return p.a.; ω its variance (smaller = more trusted)",
|
||||||
|
dataTable([
|
||||||
|
{ key: "label", label: "View" },
|
||||||
|
{ key: "q_ann", label: "q p.a.", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
{ key: "confidence", label: "Confidence", num: true, fmt: (v) => fmt.pct(v, 0) },
|
||||||
|
{ key: "omega", label: "ω", num: true, fmt: (v) => v?.toExponential(2) },
|
||||||
|
], r.views, { sortable: false }))
|
||||||
|
: null,
|
||||||
|
card("Per-stock expected returns", "implied = what the market's weights say; posterior = after the views",
|
||||||
|
dataTable([
|
||||||
|
{ key: "symbol", label: "Symbol", cls: "sym" },
|
||||||
|
{ key: "name", label: "Name", cls: "name" },
|
||||||
|
{ key: "weight", label: "Market weight", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
{ key: "vol", label: "Vol p.a.", num: true, fmt: (v) => fmt.pct(v, 1) },
|
||||||
|
{ key: "prior", label: "Implied p.a.", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
{ key: "post", label: "Posterior p.a.", num: true, fmt: (v) => fmt.pct(v, 2) },
|
||||||
|
{ key: "change", label: "Change", num: true, fmt: (v) => fmt.pctSigned(v, 2),
|
||||||
|
cls: (row) => (Math.abs(row.change) < 1e-9 ? "muted" : row.change > 0 ? "pos" : "neg") },
|
||||||
|
{ key: "sharpe", label: "Return / vol", num: true, fmt: (v) => fmt.num(v, 2) },
|
||||||
|
], rows, { sortKey: "weight", sortDir: -1 })),
|
||||||
|
);
|
||||||
|
|
||||||
|
barChart(chart, {
|
||||||
|
items: top.map((r2) => ({ label: r2.symbol, values: [r2.prior, r2.post] })),
|
||||||
|
groups: ["implied", "posterior"], valueFormat: (v) => fmt.pct(v, 0), horizontal: true, rowHeight: 24, labelWidth: 84,
|
||||||
|
});
|
||||||
|
barChart(changeChart, {
|
||||||
|
items: top.map((r2) => ({ label: r2.symbol, values: [r2.change] })),
|
||||||
|
valueFormat: (v) => fmt.pctSigned(v, 1), horizontal: true, rowHeight: 24, labelWidth: 84, diverging: true,
|
||||||
|
valueName: "change",
|
||||||
|
});
|
||||||
|
}
|
||||||
39
quantcore/api/ui/js/universe.js
Normal file
39
quantcore/api/ui/js/universe.js
Normal file
@ -0,0 +1,39 @@
|
|||||||
|
/* The investable universe, fetched once and filtered in the browser (745 rows). */
|
||||||
|
|
||||||
|
import { api } from "./core.js";
|
||||||
|
|
||||||
|
let cache = null;
|
||||||
|
|
||||||
|
export async function universe() {
|
||||||
|
if (!cache) {
|
||||||
|
const raw = await api.get("/data/symbols", { limit: 2000 });
|
||||||
|
const bySymbol = new Map(raw.symbols.map((s) => [s.symbol, s]));
|
||||||
|
cache = { ...raw, rows: raw.symbols, bySymbol };
|
||||||
|
}
|
||||||
|
return cache;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function clearUniverse() { cache = null; }
|
||||||
|
|
||||||
|
export function info(symbol) { return cache?.bySymbol.get(symbol) || null; }
|
||||||
|
|
||||||
|
export function name(symbol) { return info(symbol)?.name || symbol; }
|
||||||
|
|
||||||
|
export function sector(symbol) { return info(symbol)?.sector || "Unknown"; }
|
||||||
|
|
||||||
|
export function price(symbol) { return info(symbol)?.price ?? null; }
|
||||||
|
|
||||||
|
/** Filter the cached rows. `{region, sector, country, search}` -- all optional, all case-insensitive. */
|
||||||
|
export function filterRows(rows, f = {}) {
|
||||||
|
const q = (f.search || "").trim().toLowerCase();
|
||||||
|
return rows.filter((r) =>
|
||||||
|
(!f.region || r.region === f.region) &&
|
||||||
|
(!f.sector || (r.sector || "Unknown") === f.sector) &&
|
||||||
|
(!f.country || r.country === f.country) &&
|
||||||
|
(!q || r.symbol.toLowerCase().includes(q) || (r.name || "").toLowerCase().includes(q)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The `n` largest by market capitalisation under the filter. */
|
||||||
|
export function topSymbols(rows, f, n) {
|
||||||
|
return filterRows(rows, f).slice(0, n).map((r) => r.symbol);
|
||||||
|
}
|
||||||
254
quantcore/api/ui/styles.css
Normal file
254
quantcore/api/ui/styles.css
Normal file
@ -0,0 +1,254 @@
|
|||||||
|
/* quantcore test bench -- design tokens and layout.
|
||||||
|
Colours: the validated reference palette (categorical slots 1-8, a blue sequential ramp and a
|
||||||
|
blue<->red diverging pair with a neutral grey midpoint). Dark mode is a selected set of steps
|
||||||
|
for the dark surface, not an automatic flip. */
|
||||||
|
|
||||||
|
:root {
|
||||||
|
color-scheme: light;
|
||||||
|
--surface-0: #f2f1ee; /* page */
|
||||||
|
--surface-1: #fcfcfb; /* card / chart surface */
|
||||||
|
--surface-2: #f6f5f2; /* inset: table heads, wells */
|
||||||
|
--border: #e3e2dd;
|
||||||
|
--border-strong: #cfcec8;
|
||||||
|
--text-primary: #14130f;
|
||||||
|
--text-secondary: #52514e;
|
||||||
|
--text-muted: #85837c;
|
||||||
|
|
||||||
|
--series-1: #2a78d6; --series-2: #eb6834; --series-3: #1baf7a; --series-4: #eda100;
|
||||||
|
--series-5: #e87ba4; --series-6: #008300; --series-7: #4a3aa7; --series-8: #e34948;
|
||||||
|
|
||||||
|
--seq-100: #cde2fb; --seq-250: #86b6ef; --seq-400: #3987e5; --seq-550: #1c5cab; --seq-700: #0d366b;
|
||||||
|
--div-pos: #2a78d6; /* diverging: cool pole */
|
||||||
|
--div-neg: #e34948; /* diverging: warm pole */
|
||||||
|
--div-mid: #f0efec; /* neutral midpoint */
|
||||||
|
|
||||||
|
--good: #0f7a4a; --warning: #b26a00; --serious: #c2410c; --critical: #c02626;
|
||||||
|
--accent: #2a78d6;
|
||||||
|
--shadow: 0 1px 2px rgba(20, 19, 15, .05), 0 4px 16px rgba(20, 19, 15, .05);
|
||||||
|
--radius: 10px;
|
||||||
|
--mono: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace;
|
||||||
|
--sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Inter, Roboto, Helvetica, Arial, sans-serif;
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (prefers-color-scheme: dark) {
|
||||||
|
:root:where(:not([data-theme="light"])) {
|
||||||
|
color-scheme: dark;
|
||||||
|
--surface-0: #121211; --surface-1: #1a1a19; --surface-2: #232321;
|
||||||
|
--border: #333331; --border-strong: #46453f;
|
||||||
|
--text-primary: #ffffff; --text-secondary: #c3c2b7; --text-muted: #8f8e85;
|
||||||
|
--series-1: #3987e5; --series-2: #d95926; --series-3: #199e70; --series-4: #c98500;
|
||||||
|
--series-5: #d55181; --series-6: #008300; --series-7: #9085e9; --series-8: #e66767;
|
||||||
|
--seq-100: #0d366b; --seq-250: #184f95; --seq-400: #256abf; --seq-550: #3987e5; --seq-700: #9ec5f4;
|
||||||
|
--div-pos: #3987e5; --div-neg: #e66767; --div-mid: #383835;
|
||||||
|
--good: #37b57d; --warning: #d9a02c; --serious: #e07a3c; --critical: #e66767;
|
||||||
|
--accent: #3987e5;
|
||||||
|
--shadow: 0 1px 2px rgba(0, 0, 0, .4), 0 4px 16px rgba(0, 0, 0, .3);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
:root[data-theme="dark"] {
|
||||||
|
color-scheme: dark;
|
||||||
|
--surface-0: #121211; --surface-1: #1a1a19; --surface-2: #232321;
|
||||||
|
--border: #333331; --border-strong: #46453f;
|
||||||
|
--text-primary: #ffffff; --text-secondary: #c3c2b7; --text-muted: #8f8e85;
|
||||||
|
--series-1: #3987e5; --series-2: #d95926; --series-3: #199e70; --series-4: #c98500;
|
||||||
|
--series-5: #d55181; --series-6: #008300; --series-7: #9085e9; --series-8: #e66767;
|
||||||
|
--seq-100: #0d366b; --seq-250: #184f95; --seq-400: #256abf; --seq-550: #3987e5; --seq-700: #9ec5f4;
|
||||||
|
--div-pos: #3987e5; --div-neg: #e66767; --div-mid: #383835;
|
||||||
|
--good: #37b57d; --warning: #d9a02c; --serious: #e07a3c; --critical: #e66767;
|
||||||
|
--accent: #3987e5;
|
||||||
|
--shadow: 0 1px 2px rgba(0, 0, 0, .4), 0 4px 16px rgba(0, 0, 0, .3);
|
||||||
|
}
|
||||||
|
|
||||||
|
* { box-sizing: border-box; }
|
||||||
|
html, body { height: 100%; }
|
||||||
|
body {
|
||||||
|
margin: 0; background: var(--surface-0); color: var(--text-primary);
|
||||||
|
font-family: var(--sans); font-size: 13px; line-height: 1.45;
|
||||||
|
-webkit-font-smoothing: antialiased;
|
||||||
|
}
|
||||||
|
h1, h2, h3, h4 { margin: 0; font-weight: 600; letter-spacing: -.01em; }
|
||||||
|
h2 { font-size: 15px; }
|
||||||
|
h3 { font-size: 13px; }
|
||||||
|
a { color: var(--accent); }
|
||||||
|
|
||||||
|
/* ---------------------------------------------------------------- app frame */
|
||||||
|
.app { display: flex; flex-direction: column; min-height: 100vh; }
|
||||||
|
|
||||||
|
.topbar {
|
||||||
|
display: flex; align-items: center; gap: 16px; padding: 10px 18px;
|
||||||
|
background: var(--surface-1); border-bottom: 1px solid var(--border); position: sticky; top: 0; z-index: 30;
|
||||||
|
}
|
||||||
|
.brand { display: flex; align-items: baseline; gap: 9px; }
|
||||||
|
.brand b { font-size: 15px; letter-spacing: -.02em; }
|
||||||
|
.brand span { color: var(--text-muted); font-size: 11px; }
|
||||||
|
.status-row { display: flex; gap: 6px; flex-wrap: wrap; margin-left: auto; align-items: center; }
|
||||||
|
|
||||||
|
.pill {
|
||||||
|
display: inline-flex; align-items: center; gap: 6px; padding: 3px 9px; border-radius: 999px;
|
||||||
|
background: var(--surface-2); border: 1px solid var(--border); font-size: 11px; color: var(--text-secondary);
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
.pill b { color: var(--text-primary); font-weight: 600; }
|
||||||
|
.dot { width: 7px; height: 7px; border-radius: 50%; background: var(--text-muted); flex: none; }
|
||||||
|
.dot.ok { background: var(--good); } .dot.warn { background: var(--warning); } .dot.bad { background: var(--critical); }
|
||||||
|
|
||||||
|
.tabs { display: flex; gap: 2px; padding: 0 18px; background: var(--surface-1); border-bottom: 1px solid var(--border);
|
||||||
|
position: sticky; top: 49px; z-index: 29; overflow-x: auto; }
|
||||||
|
.tabs button {
|
||||||
|
appearance: none; background: none; border: 0; border-bottom: 2px solid transparent; cursor: pointer;
|
||||||
|
padding: 9px 13px; font: inherit; font-weight: 500; color: var(--text-secondary); white-space: nowrap;
|
||||||
|
}
|
||||||
|
.tabs button:hover { color: var(--text-primary); }
|
||||||
|
.tabs button[aria-selected="true"] { color: var(--text-primary); border-bottom-color: var(--accent); }
|
||||||
|
|
||||||
|
.body { display: flex; align-items: flex-start; gap: 0; flex: 1; min-height: 0; }
|
||||||
|
.sidebar {
|
||||||
|
width: 302px; flex: none; background: var(--surface-1); border-right: 1px solid var(--border);
|
||||||
|
padding: 14px; position: sticky; top: 84px; max-height: calc(100vh - 84px); overflow-y: auto;
|
||||||
|
}
|
||||||
|
.main { flex: 1; min-width: 0; padding: 16px 18px 48px; }
|
||||||
|
.tabpanel[hidden] { display: none; }
|
||||||
|
|
||||||
|
/* ---------------------------------------------------------------- cards */
|
||||||
|
.card {
|
||||||
|
background: var(--surface-1); border: 1px solid var(--border); border-radius: var(--radius);
|
||||||
|
padding: 14px; margin-bottom: 14px; box-shadow: var(--shadow);
|
||||||
|
}
|
||||||
|
.card > header { display: flex; align-items: baseline; gap: 10px; margin-bottom: 10px; flex-wrap: wrap; }
|
||||||
|
.card > header .sub { color: var(--text-muted); font-size: 11px; }
|
||||||
|
.card > header .spacer { margin-left: auto; }
|
||||||
|
.grid { display: grid; gap: 14px; }
|
||||||
|
.grid.two { grid-template-columns: repeat(auto-fit, minmax(380px, 1fr)); align-items: start; }
|
||||||
|
.grid.three { grid-template-columns: repeat(auto-fit, minmax(260px, 1fr)); }
|
||||||
|
@media (max-width: 900px) { .body { flex-direction: column; } .sidebar { width: 100%; position: static; max-height: none; border-right: 0; border-bottom: 1px solid var(--border); } }
|
||||||
|
|
||||||
|
/* ---------------------------------------------------------------- forms */
|
||||||
|
.field { display: flex; flex-direction: column; gap: 3px; margin-bottom: 9px; min-width: 0; }
|
||||||
|
.field > label { font-size: 11px; color: var(--text-secondary); font-weight: 500; }
|
||||||
|
.field .hint { font-size: 10.5px; color: var(--text-muted); }
|
||||||
|
input, select, textarea, button { font: inherit; font-size: 12.5px; color: var(--text-primary); }
|
||||||
|
input[type="text"], input[type="number"], input[type="date"], input[type="search"], select, textarea {
|
||||||
|
background: var(--surface-1); border: 1px solid var(--border-strong); border-radius: 7px;
|
||||||
|
padding: 5px 8px; width: 100%; min-width: 0;
|
||||||
|
}
|
||||||
|
input:focus-visible, select:focus-visible, textarea:focus-visible, button:focus-visible {
|
||||||
|
outline: 2px solid var(--accent); outline-offset: 1px;
|
||||||
|
}
|
||||||
|
input[type="number"] { font-variant-numeric: tabular-nums; }
|
||||||
|
.row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; }
|
||||||
|
.row > .field { flex: 1; }
|
||||||
|
.row > .field.auto { flex: 0 0 auto; }
|
||||||
|
.row > .field.auto > .btnbar { height: 27px; }
|
||||||
|
.check { display: flex; align-items: center; gap: 6px; font-size: 12px; color: var(--text-secondary); }
|
||||||
|
.check input { width: auto; }
|
||||||
|
|
||||||
|
button.btn {
|
||||||
|
background: var(--surface-2); border: 1px solid var(--border-strong); border-radius: 7px;
|
||||||
|
padding: 5px 11px; cursor: pointer; font-weight: 500; white-space: nowrap;
|
||||||
|
}
|
||||||
|
button.btn:hover { border-color: var(--text-muted); }
|
||||||
|
button.btn.primary { background: var(--accent); border-color: var(--accent); color: #fff; }
|
||||||
|
button.btn.primary:hover { filter: brightness(1.07); }
|
||||||
|
button.btn.small { padding: 2px 7px; font-size: 11px; }
|
||||||
|
button.btn:disabled { opacity: .5; cursor: not-allowed; }
|
||||||
|
.btnbar { display: flex; gap: 7px; flex-wrap: wrap; align-items: center; }
|
||||||
|
|
||||||
|
/* sidebar sections */
|
||||||
|
details.section { border-top: 1px solid var(--border); padding: 9px 0; }
|
||||||
|
details.section:first-of-type { border-top: 0; }
|
||||||
|
details.section > summary {
|
||||||
|
cursor: pointer; font-weight: 600; font-size: 12px; list-style: none; display: flex; align-items: center; gap: 7px;
|
||||||
|
color: var(--text-primary); padding: 2px 0;
|
||||||
|
}
|
||||||
|
details.section > summary::-webkit-details-marker { display: none; }
|
||||||
|
details.section > summary::before {
|
||||||
|
content: ""; width: 0; height: 0; border-left: 4px solid var(--text-muted);
|
||||||
|
border-top: 3.5px solid transparent; border-bottom: 3.5px solid transparent; transition: transform .12s;
|
||||||
|
}
|
||||||
|
details.section[open] > summary::before { transform: rotate(90deg); }
|
||||||
|
details.section > summary .count { margin-left: auto; color: var(--text-muted); font-weight: 400; font-size: 11px; }
|
||||||
|
details.section > div { padding-top: 9px; }
|
||||||
|
|
||||||
|
/* chips */
|
||||||
|
.chips { display: flex; flex-wrap: wrap; gap: 5px; }
|
||||||
|
.chip {
|
||||||
|
display: inline-flex; align-items: center; gap: 5px; background: var(--surface-2);
|
||||||
|
border: 1px solid var(--border); border-radius: 999px; padding: 2px 4px 2px 9px; font-size: 11px;
|
||||||
|
font-family: var(--mono);
|
||||||
|
}
|
||||||
|
.chip button { appearance: none; background: none; border: 0; cursor: pointer; color: var(--text-muted);
|
||||||
|
font-size: 13px; line-height: 1; padding: 0 3px; border-radius: 50%; }
|
||||||
|
.chip button:hover { color: var(--critical); }
|
||||||
|
|
||||||
|
/* ---------------------------------------------------------------- tables */
|
||||||
|
.table-wrap { overflow: auto; max-height: 460px; border: 1px solid var(--border); border-radius: 8px; }
|
||||||
|
table.data { border-collapse: separate; border-spacing: 0; width: 100%; font-size: 12px; }
|
||||||
|
table.data th, table.data td { padding: 5px 9px; text-align: left; white-space: nowrap; border-bottom: 1px solid var(--border); }
|
||||||
|
table.data thead th {
|
||||||
|
position: sticky; top: 0; background: var(--surface-2); z-index: 2; font-weight: 600; font-size: 11px;
|
||||||
|
color: var(--text-secondary); cursor: default; border-bottom: 1px solid var(--border-strong);
|
||||||
|
}
|
||||||
|
table.data thead th.sortable { cursor: pointer; user-select: none; }
|
||||||
|
table.data thead th.sortable:hover { color: var(--text-primary); }
|
||||||
|
table.data thead th .arrow { color: var(--text-muted); font-size: 9px; }
|
||||||
|
table.data td.num, table.data th.num { text-align: right; font-variant-numeric: tabular-nums; font-family: var(--mono); font-size: 11.5px; }
|
||||||
|
table.data tbody tr:hover td { background: var(--surface-2); }
|
||||||
|
/* the row whose result is on screen: an accent edge plus a tint, so it reads without colour alone */
|
||||||
|
table.data tbody tr.selected td { background: var(--surface-2); font-weight: 600; }
|
||||||
|
table.data tbody tr.selected td:first-child { box-shadow: inset 3px 0 0 var(--accent); }
|
||||||
|
table.data tbody tr:last-child td { border-bottom: 0; }
|
||||||
|
table.data td.sym { font-family: var(--mono); font-weight: 600; }
|
||||||
|
table.data td.name { max-width: 230px; overflow: hidden; text-overflow: ellipsis; }
|
||||||
|
.pos { color: var(--good); } .neg { color: var(--critical); }
|
||||||
|
.muted { color: var(--text-muted); }
|
||||||
|
.mono { font-family: var(--mono); }
|
||||||
|
|
||||||
|
/* ---------------------------------------------------------------- stat tiles */
|
||||||
|
.stats { display: grid; grid-template-columns: repeat(auto-fit, minmax(132px, 1fr)); gap: 1px;
|
||||||
|
background: var(--border); border: 1px solid var(--border); border-radius: 8px; overflow: hidden; }
|
||||||
|
.stat { background: var(--surface-1); padding: 9px 11px; }
|
||||||
|
.stat .label { font-size: 10.5px; color: var(--text-secondary); }
|
||||||
|
.stat .value { font-size: 17px; font-weight: 600; font-variant-numeric: tabular-nums; letter-spacing: -.02em; margin-top: 1px; }
|
||||||
|
.stat .delta { font-size: 10.5px; font-variant-numeric: tabular-nums; }
|
||||||
|
.hero { font-size: 30px; font-weight: 600; letter-spacing: -.03em; font-variant-numeric: tabular-nums; }
|
||||||
|
|
||||||
|
/* ---------------------------------------------------------------- misc */
|
||||||
|
.chart { width: 100%; min-width: 0; }
|
||||||
|
.chart svg { margin: 0 auto; }
|
||||||
|
.chart svg { display: block; width: 100%; height: auto; overflow: visible; }
|
||||||
|
.legend { display: flex; flex-wrap: wrap; gap: 4px 14px; margin: 6px 0 2px; font-size: 11px; color: var(--text-secondary); }
|
||||||
|
.legend .item { display: inline-flex; align-items: center; gap: 5px; cursor: default; }
|
||||||
|
.legend .key { width: 12px; height: 2px; border-radius: 1px; flex: none; }
|
||||||
|
.legend .key.rect { height: 9px; width: 9px; border-radius: 2px; }
|
||||||
|
|
||||||
|
#tooltip {
|
||||||
|
position: fixed; z-index: 100; pointer-events: none; opacity: 0; transition: opacity .08s;
|
||||||
|
background: var(--surface-1); border: 1px solid var(--border-strong); border-radius: 7px;
|
||||||
|
box-shadow: var(--shadow); padding: 7px 9px; font-size: 11.5px; max-width: 280px;
|
||||||
|
}
|
||||||
|
#tooltip .tt-title { font-weight: 600; margin-bottom: 3px; }
|
||||||
|
#tooltip .tt-row { display: flex; align-items: center; gap: 7px; }
|
||||||
|
#tooltip .tt-row .key { width: 10px; height: 2px; border-radius: 1px; flex: none; }
|
||||||
|
#tooltip .tt-row .nm { color: var(--text-secondary); }
|
||||||
|
#tooltip .tt-row .v { margin-left: auto; font-variant-numeric: tabular-nums; font-weight: 600; }
|
||||||
|
|
||||||
|
.note { border-left: 2px solid var(--border-strong); padding: 2px 0 2px 9px; color: var(--text-secondary); font-size: 11.5px; }
|
||||||
|
.warn-box { border-left: 2px solid var(--warning); background: var(--surface-2); padding: 7px 10px; border-radius: 0 7px 7px 0;
|
||||||
|
font-size: 11.5px; color: var(--text-secondary); margin-bottom: 8px; }
|
||||||
|
.error-box { border-left: 2px solid var(--critical); background: var(--surface-2); padding: 7px 10px; border-radius: 0 7px 7px 0;
|
||||||
|
font-size: 11.5px; color: var(--text-primary); margin-bottom: 8px; white-space: pre-wrap; }
|
||||||
|
.kv { display: grid; grid-template-columns: max-content 1fr; gap: 2px 12px; font-size: 11.5px; }
|
||||||
|
.kv dt { color: var(--text-secondary); }
|
||||||
|
.kv dd { margin: 0; font-variant-numeric: tabular-nums; overflow-wrap: anywhere; min-width: 0; }
|
||||||
|
.empty { color: var(--text-muted); font-size: 12px; padding: 18px; text-align: center; border: 1px dashed var(--border-strong); border-radius: 8px; }
|
||||||
|
.busy { opacity: .45; transition: opacity .1s; }
|
||||||
|
.spinner { width: 12px; height: 12px; border: 2px solid var(--border-strong); border-top-color: var(--accent);
|
||||||
|
border-radius: 50%; animation: spin .7s linear infinite; display: inline-block; vertical-align: -2px; }
|
||||||
|
@keyframes spin { to { transform: rotate(360deg); } }
|
||||||
|
.seg { display: inline-flex; align-self: flex-start; flex: none; border: 1px solid var(--border-strong);
|
||||||
|
border-radius: 7px; overflow: hidden; }
|
||||||
|
.seg button { appearance: none; flex: none; background: var(--surface-1); border: 0; border-left: 1px solid var(--border);
|
||||||
|
padding: 4px 10px; cursor: pointer; color: var(--text-secondary); font-size: 12px; }
|
||||||
|
.seg button:first-child { border-left: 0; }
|
||||||
|
.seg button[aria-pressed="true"] { background: var(--accent); color: #fff; }
|
||||||
|
.tabbar-sub { display: flex; gap: 4px; margin-bottom: 10px; flex-wrap: wrap; }
|
||||||
672
quantcore/apicli.py
Normal file
672
quantcore/apicli.py
Normal file
@ -0,0 +1,672 @@
|
|||||||
|
"""Command-line client of the quant-core API: ``quantcore-api <command>``.
|
||||||
|
|
||||||
|
``quantcore`` (the other entry point) runs the services *in this process* and needs the data
|
||||||
|
directory locally. ``quantcore-api`` talks to a *running* service over HTTP and needs only its URL,
|
||||||
|
so it works from any machine on the network and is the quickest way to see what the API returns --
|
||||||
|
and, with ``selftest``, to check that a deployment answers correctly on every endpoint.
|
||||||
|
|
||||||
|
quantcore-api --url http://spark-6d4a:5001 health
|
||||||
|
quantcore-api selftest # exercise every endpoint, pass/fail table
|
||||||
|
quantcore-api endpoints # the API surface, from its own OpenAPI schema
|
||||||
|
|
||||||
|
quantcore-api universe --region Europe --sector "Health Care"
|
||||||
|
quantcore-api prices NESN.SW AAPL --freq weekly --tail 5
|
||||||
|
quantcore-api risk NESN.SW AAPL --horizon week --drift zero
|
||||||
|
quantcore-api correlation NESN.SW NOVN.SW ROG.SW
|
||||||
|
quantcore-api expected-returns --max-assets 50 --view AAPL:0.6:0.4
|
||||||
|
quantcore-api scenarios NESN.SW AAPL --model gbm --n-scenarios 4096
|
||||||
|
quantcore-api optimize --cash 1000000 --cvar-limit 40000 --horizon week --risk-drift zero
|
||||||
|
quantcore-api optimize --request my_request.json --json
|
||||||
|
quantcore-api update --wait # start the data-refresh job and follow it
|
||||||
|
|
||||||
|
The base URL comes from ``--url``, else ``QUANTCORE_API_URL``, else ``http://127.0.0.1:5001``.
|
||||||
|
Every command takes ``--json`` to print the raw answer instead of a table; that is the form to pipe
|
||||||
|
into ``jq`` or to save as a fixture.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import pathlib
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from quantcore.client import QuantcoreClient, QuantcoreError, default_url
|
||||||
|
|
||||||
|
WIDE = {"display.width": 200, "display.max_columns": 40, "display.max_rows": 200}
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# output helpers
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
def _json(obj: Any) -> int:
|
||||||
|
print(json.dumps(obj, indent=2, default=str))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def _records(rows: list[dict], transpose: bool = False, round_to: int = 4) -> int:
|
||||||
|
"""A list of JSON objects as a table; transposed when the rows are wide (risk tables)."""
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
if not rows:
|
||||||
|
print("(no rows)")
|
||||||
|
return 0
|
||||||
|
df = pd.DataFrame(rows)
|
||||||
|
if transpose:
|
||||||
|
df = df.set_index(df.columns[0]).T
|
||||||
|
with pd.option_context(*[x for kv in WIDE.items() for x in kv]):
|
||||||
|
print(df.round(round_to).to_string())
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def _frame(body: dict, tail: int | None = None) -> int:
|
||||||
|
"""A ``{"index", "columns", "data"}`` answer as a table, optionally only its last rows."""
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
df = pd.DataFrame(body.get("data") or [], index=body.get("index") or [], columns=body.get("columns") or [])
|
||||||
|
head = {k: v for k, v in body.items() if k not in ("index", "columns", "data")}
|
||||||
|
if head:
|
||||||
|
print(" · ".join(f"{k}={v}" for k, v in head.items()))
|
||||||
|
print(f"{len(df)} rows x {len(df.columns)} columns" + (f", last {tail}" if tail and len(df) > tail else ""))
|
||||||
|
if tail:
|
||||||
|
df = df.tail(tail)
|
||||||
|
with pd.option_context(*[x for kv in WIDE.items() for x in kv]):
|
||||||
|
print(df.round(6).to_string())
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def _matrix(body: dict, key: str) -> int:
|
||||||
|
inner = body.get(key, body)
|
||||||
|
return _frame(inner if isinstance(inner, dict) and "data" in inner else body)
|
||||||
|
|
||||||
|
|
||||||
|
def _client(args: argparse.Namespace) -> QuantcoreClient:
|
||||||
|
return QuantcoreClient(args.url, timeout=args.timeout)
|
||||||
|
|
||||||
|
|
||||||
|
def _views(specs: list[str] | None) -> list[dict]:
|
||||||
|
"""``--view SYMBOL:SENTIMENT[:CONFIDENCE[:STRENGTH]]`` -> the API's view objects.
|
||||||
|
|
||||||
|
``--view AAPL:0.6:0.4`` reads "positive news on Apple (sentiment 0.6), moderate confidence".
|
||||||
|
"""
|
||||||
|
views = []
|
||||||
|
for spec in specs or []:
|
||||||
|
parts = spec.split(":")
|
||||||
|
if len(parts) < 2:
|
||||||
|
raise SystemExit(f"--view needs SYMBOL:SENTIMENT[:CONFIDENCE[:STRENGTH]], got {spec!r}")
|
||||||
|
view: dict[str, Any] = {"symbol": parts[0], "sentiment": float(parts[1])}
|
||||||
|
if len(parts) > 2:
|
||||||
|
view["confidence"] = float(parts[2])
|
||||||
|
if len(parts) > 3:
|
||||||
|
view["strength"] = float(parts[3])
|
||||||
|
views.append(view)
|
||||||
|
return views
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# commands
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
def cmd_health(args) -> int:
|
||||||
|
body = _client(args).health()
|
||||||
|
if args.json:
|
||||||
|
return _json(body)
|
||||||
|
store, cuopt = body.get("store") or {}, body.get("cuopt") or {}
|
||||||
|
print(f"{args.url or default_url()} version {body.get('version')} currency {body.get('base_currency')}")
|
||||||
|
print(f"store : {store.get('n_universe')} stocks in the universe, {store.get('n_symbols')} with prices, "
|
||||||
|
f"{store.get('n_days')} days, last {store.get('last_price_date')}")
|
||||||
|
print(f"cuOpt : {'reachable, ' + str(cuopt.get('version')) if cuopt.get('reachable') else 'NOT reachable'}"
|
||||||
|
f"{'' if cuopt.get('reachable') else ' - optimisations fall back to HiGHS on the CPU'}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_endpoints(args) -> int:
|
||||||
|
schema = _client(args).openapi()
|
||||||
|
rows = [{"method": method.upper(), "path": path, "summary": op.get("summary") or op.get("operationId", ""),
|
||||||
|
"tag": (op.get("tags") or [""])[0]}
|
||||||
|
for path, ops in sorted((schema.get("paths") or {}).items())
|
||||||
|
for method, op in ops.items() if method in ("get", "post", "put", "delete")]
|
||||||
|
if args.json:
|
||||||
|
return _json(rows)
|
||||||
|
print(f"{schema.get('info', {}).get('title', 'quantcore')} -- {len(rows)} endpoints "
|
||||||
|
f"(interactive docs at {(args.url or default_url()).rstrip('/')}/docs)")
|
||||||
|
return _records(rows, round_to=0)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_universe(args) -> int:
|
||||||
|
rows = _client(args).universe(region=args.region, sector=args.sector, country=args.country)
|
||||||
|
if args.json:
|
||||||
|
return _json(rows)
|
||||||
|
keep = ["symbol", "name", "region", "country", "gics_sector", "listing_currency", "market_cap_base"]
|
||||||
|
rows = sorted(rows, key=lambda r: -(r.get("market_cap_base") or 0))[: args.limit]
|
||||||
|
print(f"{len(rows)} of the matching stocks, largest first")
|
||||||
|
return _records([{k: r.get(k) for k in keep} for r in rows], round_to=0)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_symbol(args) -> int:
|
||||||
|
return _json(_client(args).symbol(args.symbol))
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_prices(args) -> int:
|
||||||
|
body = _client(args).prices(args.symbols, start=args.start, end=args.end, currency=args.currency,
|
||||||
|
freq=args.freq)
|
||||||
|
return _json(body) if args.json else _frame(body, args.tail)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_returns(args) -> int:
|
||||||
|
body = _client(args).returns(args.symbols, start=args.start, end=args.end, currency=args.currency,
|
||||||
|
freq=args.freq)
|
||||||
|
return _json(body) if args.json else _frame(body, args.tail)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_market(args) -> int:
|
||||||
|
qc = _client(args)
|
||||||
|
if args.what == "weights":
|
||||||
|
body = qc.market_weights(max_assets=args.max_assets)
|
||||||
|
if args.json:
|
||||||
|
return _json(body)
|
||||||
|
weights = body.get("weights") or {}
|
||||||
|
print(f"as of {body.get('as_of')}, {len(weights)} stocks, sum {sum(weights.values()):.6f}")
|
||||||
|
return _records([{"symbol": k, "weight": v} for k, v in list(weights.items())[: args.limit]], round_to=6)
|
||||||
|
body = qc.market_index(start=args.start, end=args.end, freq=args.freq)
|
||||||
|
if args.json:
|
||||||
|
return _json(body)
|
||||||
|
print(f"cap-weighted index, {body.get('freq')}: mean_ann {body.get('mean_ann'):.4%}, "
|
||||||
|
f"vol_ann {body.get('vol_ann'):.4%}, {body.get('n')} observations")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_risk(args) -> int:
|
||||||
|
rows = _client(args).asset_risk(args.symbols, horizon=args.horizon, freq=args.freq, window=args.window,
|
||||||
|
lam=args.lam, currency=args.currency, drift=args.drift, space=args.space,
|
||||||
|
confidence=args.confidence)
|
||||||
|
if args.json:
|
||||||
|
return _json(rows)
|
||||||
|
print(f"horizon {args.horizon} · {args.freq} · window {args.window} · lambda {args.lam} · "
|
||||||
|
f"drift {args.drift} · space {args.space}")
|
||||||
|
return _records(rows, transpose=True, round_to=6)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_risk_portfolio(args) -> int:
|
||||||
|
body = _client(args).portfolio_risk(json.loads(args.holdings), cash=args.cash, horizon=args.horizon,
|
||||||
|
method=args.method, confidence=args.confidence)
|
||||||
|
return _json(body)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_matrix(args) -> int:
|
||||||
|
qc = _client(args)
|
||||||
|
call = qc.correlation if args.command == "correlation" else qc.covariance
|
||||||
|
body = call(args.symbols, method=args.method, window=args.window, lam=args.lam, freq=args.freq)
|
||||||
|
return _json(body) if args.json else _frame(body)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_performance(args) -> int:
|
||||||
|
rows = _client(args).asset_performance(args.symbols, freq=args.freq, currency=args.currency, start=args.start)
|
||||||
|
return _json(rows) if args.json else _records(rows, transpose=True)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_expected_returns(args) -> int:
|
||||||
|
body = _client(args).expected_returns(max_assets=args.max_assets, symbols=args.symbols, tau=args.tau,
|
||||||
|
years=args.years, omega_method=args.omega_method,
|
||||||
|
views=_views(args.view))
|
||||||
|
if args.json:
|
||||||
|
return _json(body)
|
||||||
|
rows = body.get("expected_returns") or body.get("returns") or []
|
||||||
|
for k, v in body.items():
|
||||||
|
if isinstance(v, (int, float, str)):
|
||||||
|
print(f"{k:24} {v}")
|
||||||
|
if isinstance(rows, dict):
|
||||||
|
rows = [{"symbol": k, **(v if isinstance(v, dict) else {"value": v})} for k, v in rows.items()]
|
||||||
|
return _records(rows[: args.limit] if isinstance(rows, list) else [], round_to=6)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_model_covariance(args) -> int:
|
||||||
|
body = _client(args).model_covariance(args.symbols, freq=args.freq, horizon=args.horizon,
|
||||||
|
covariance=_cov_spec(args))
|
||||||
|
if args.json:
|
||||||
|
return _json(body)
|
||||||
|
print(f"model {body.get('model')} · as of {body.get('as_of')} · shrinkage applied "
|
||||||
|
f"{body.get('shrinkage_applied')}")
|
||||||
|
print("conditioning:", json.dumps(body.get("conditioning"), default=str))
|
||||||
|
return _matrix(body, "covariance")
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_scenarios(args) -> int:
|
||||||
|
body = _client(args).scenarios(args.symbols, model=args.model, n_scenarios=args.n_scenarios,
|
||||||
|
horizon=args.horizon, freq=args.freq, seed=args.seed,
|
||||||
|
sampling=args.sampling, covariance=_cov_spec(args),
|
||||||
|
expected_returns=args.expected_returns,
|
||||||
|
importance_sampling=args.importance_sampling,
|
||||||
|
holdings=json.loads(args.holdings) if args.holdings else None,
|
||||||
|
value=args.value, confidence=args.confidence)
|
||||||
|
if args.json:
|
||||||
|
return _json(body)
|
||||||
|
desc, port = body.get("description") or {}, body.get("portfolio") or {}
|
||||||
|
print(f"{desc.get('model')} · {desc.get('n_scenarios')} scenarios x {desc.get('n_assets')} assets · "
|
||||||
|
f"horizon {desc.get('horizon_periods')} periods · seed {desc.get('seed')}")
|
||||||
|
print(f"hash {desc.get('hash')} · shocks {desc.get('shock_hash')} · weighted {desc.get('weighted')} · "
|
||||||
|
f"effective sample size {desc.get('effective_sample_size')}")
|
||||||
|
if port:
|
||||||
|
print(f"portfolio: invested {port.get('invested'):,.0f}")
|
||||||
|
for level, fig in sorted((port.get("risk") or {}).items()):
|
||||||
|
print(f" {float(level):.0%} VaR {fig.get('var'):>14,.2f} CVaR {fig.get('cvar'):>14,.2f}")
|
||||||
|
return _records(body.get("summary") or [], transpose=True, round_to=6)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_optimize(args) -> int:
|
||||||
|
body: dict[str, Any] = json.loads(pathlib.Path(args.request).read_text()) if args.request else {}
|
||||||
|
flags = {"cash": args.cash, "max_assets": args.max_assets, "horizon": args.horizon,
|
||||||
|
"confidence": args.confidence, "cvar_limit": args.cvar_limit, "objective": args.objective,
|
||||||
|
"scenario_model": args.scenario_model, "n_scenarios": args.n_scenarios, "seed": args.seed,
|
||||||
|
"solver": args.solver, "time_limit": args.time_limit, "risk_drift": args.risk_drift,
|
||||||
|
"expected_returns": args.expected_returns, "importance_sampling": args.importance_sampling,
|
||||||
|
"row_generation": args.row_generation, "freq": args.freq, "symbols": args.symbols}
|
||||||
|
for k, v in flags.items(): # the request file wins over a flag default
|
||||||
|
if v is not None and k not in body:
|
||||||
|
body[k] = v
|
||||||
|
if args.holdings:
|
||||||
|
body["holdings"] = json.loads(args.holdings)
|
||||||
|
body.setdefault("mode", "rebalance")
|
||||||
|
if args.view:
|
||||||
|
body["views"] = _views(args.view)
|
||||||
|
if args.max_weight is not None or args.max_sector_weight is not None:
|
||||||
|
constraints = dict(body.get("constraints") or {})
|
||||||
|
if args.max_weight is not None:
|
||||||
|
constraints["max_weight"] = args.max_weight
|
||||||
|
if args.max_sector_weight is not None:
|
||||||
|
constraints["max_sector_weight"] = args.max_sector_weight
|
||||||
|
body["constraints"] = constraints
|
||||||
|
|
||||||
|
t0 = time.monotonic()
|
||||||
|
res = _client(args).optimize(**body)
|
||||||
|
elapsed = time.monotonic() - t0
|
||||||
|
if args.json:
|
||||||
|
return _json(res)
|
||||||
|
print(f"status {res.get('status')} · solver {res.get('solver')} · reproducibility "
|
||||||
|
f"{res.get('reproducibility')} · {elapsed:.1f} s wall clock")
|
||||||
|
info = res.get("solver_info") or {}
|
||||||
|
if info.get("mip_gap") is not None:
|
||||||
|
print(f"mip gap {info['mip_gap']:.2e}" + (f" · {info.get('message')}" if info.get("message") else ""))
|
||||||
|
key = (res.get("scenario_key") or {}).get("id")
|
||||||
|
print(f"scenario key {key} · hash {res.get('scenario_hash')} · timings {res.get('timings')}")
|
||||||
|
print(f"wealth {res.get('wealth', 0):,.0f} · invested {res.get('invested', 0):,.0f} · "
|
||||||
|
f"cash left {res.get('cash_end', 0):,.0f} · cost {res.get('total_cost', 0):,.0f} · "
|
||||||
|
f"turnover {res.get('turnover', 0):,.0f}")
|
||||||
|
print(f"E[P&L] {res.get('expected_pnl', 0):,.0f} ({res.get('expected_return', 0):.3%}) · "
|
||||||
|
f"VaR {res.get('var', 0):,.0f} · CVaR {res.get('cvar', 0):,.0f} "
|
||||||
|
f"(limit {res.get('cvar_limit')}) at {res.get('confidence', 0):.0%} over {res.get('horizon')}")
|
||||||
|
for warning in res.get("warnings") or []:
|
||||||
|
print(f" ! {warning}")
|
||||||
|
positions = res.get("positions") or []
|
||||||
|
if positions and not args.quiet:
|
||||||
|
keep = [k for k in ("symbol", "name", "sector", "price", "shares", "trade", "value", "weight")
|
||||||
|
if k in positions[0]]
|
||||||
|
_records([{k: p.get(k) for k in keep} for p in positions], round_to=3)
|
||||||
|
print("sectors:", {k: round(v, 3) for k, v in (res.get("sector_exposure") or {}).items()})
|
||||||
|
return 0 if res.get("status") in ("Optimal", "FeasibleFound") else 2
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_jobs(args) -> int:
|
||||||
|
qc = _client(args)
|
||||||
|
if args.job_id:
|
||||||
|
return _json(qc.job(args.job_id))
|
||||||
|
return _json(qc.jobs())
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_setup(args) -> int:
|
||||||
|
return _start_job(args, _client(args).start_setup(offline=args.offline, max_per_region=args.max_per_region))
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_update(args) -> int:
|
||||||
|
return _start_job(args, _client(args).start_update(full=args.full))
|
||||||
|
|
||||||
|
|
||||||
|
def _start_job(args, started: dict) -> int:
|
||||||
|
job_id = started.get("job_id") or started.get("id")
|
||||||
|
print(f"job {job_id} started ({started.get('status', 'queued')})")
|
||||||
|
if not args.wait:
|
||||||
|
print(f"follow it with: quantcore-api --url {args.url or default_url()} jobs --job-id {job_id}")
|
||||||
|
return 0
|
||||||
|
record = _client(args).wait_for_job(job_id, poll=args.poll,
|
||||||
|
on_poll=lambda r: print(f" ... {r.get('status')}", flush=True))
|
||||||
|
_json(record)
|
||||||
|
return 0 if record.get("status") == "done" else 2
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_snapshots(args) -> int:
|
||||||
|
qc = _client(args)
|
||||||
|
if args.create:
|
||||||
|
return _json(qc.create_snapshot(args.name))
|
||||||
|
return _json(qc.snapshots())
|
||||||
|
|
||||||
|
|
||||||
|
def _cov_spec(args) -> dict:
|
||||||
|
return {"method": args.cov_method, "lam": args.lam, "window": args.window, "shrinkage": args.shrinkage}
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# selftest -- exercise every endpoint against a running service
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
def cmd_selftest(args) -> int:
|
||||||
|
"""Call every endpoint with a small request and report what each one answered.
|
||||||
|
|
||||||
|
Read-only by default: the two background jobs and the snapshot endpoint only run with
|
||||||
|
``--write``, because they download data and write to the store. Exit code 1 if any probe failed.
|
||||||
|
"""
|
||||||
|
qc = _client(args)
|
||||||
|
results: list[dict] = []
|
||||||
|
|
||||||
|
def probe(name: str, call: Any, evidence: Any = None, skip: str = "") -> Any:
|
||||||
|
"""Run one endpoint. ``call()`` performs it, ``evidence(value)`` says what came back."""
|
||||||
|
if skip:
|
||||||
|
results.append({"endpoint": name, "result": "skipped", "ms": 0, "evidence": skip})
|
||||||
|
return None
|
||||||
|
t0 = time.monotonic()
|
||||||
|
try:
|
||||||
|
value = call()
|
||||||
|
note = evidence(value) if evidence else ""
|
||||||
|
except QuantcoreError as exc:
|
||||||
|
note, value, ok = str(exc)[:120], None, False
|
||||||
|
except Exception as exc: # noqa: BLE001 (a probe's own assertion, or a malformed answer)
|
||||||
|
note, value, ok = f"{type(exc).__name__}: {exc}"[:120], None, False
|
||||||
|
else:
|
||||||
|
ok = True
|
||||||
|
results.append({"endpoint": name, "result": "ok" if ok else "FAIL",
|
||||||
|
"ms": round((time.monotonic() - t0) * 1000), "evidence": note})
|
||||||
|
return value
|
||||||
|
|
||||||
|
def health() -> dict:
|
||||||
|
body = qc.health()
|
||||||
|
if not (body.get("store") or {}).get("prices"):
|
||||||
|
raise AssertionError("the store has no prices - run the setup job first")
|
||||||
|
return body
|
||||||
|
|
||||||
|
body = probe("GET /health", health, lambda b: (
|
||||||
|
f"{b['store'].get('n_universe')} stocks, last {b['store'].get('last_price_date')}, "
|
||||||
|
f"cuOpt {'up' if (b.get('cuopt') or {}).get('reachable') else 'down'}"))
|
||||||
|
if body is None:
|
||||||
|
_records(results, round_to=0)
|
||||||
|
print("\nthe service is not usable - stopping here")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
rows = probe("GET /universe", qc.universe, lambda r: f"{len(r)} rows") or []
|
||||||
|
syms = args.symbols or [r["symbol"] for r in
|
||||||
|
sorted(rows, key=lambda r: -(r.get("market_cap_base") or 0))[: args.n_symbols]]
|
||||||
|
if not syms:
|
||||||
|
print("no symbols to probe with")
|
||||||
|
return 1
|
||||||
|
print(f"probing with {', '.join(syms)}\n")
|
||||||
|
holdings = {syms[0]: 10}
|
||||||
|
|
||||||
|
probe("GET /openapi.json", qc.openapi, lambda d: f"{len(d.get('paths') or {})} paths")
|
||||||
|
probe("GET /universe/{symbol}", lambda: qc.symbol(syms[0]),
|
||||||
|
lambda d: f"{d.get('symbol')}: {d.get('name')}")
|
||||||
|
probe("GET /data/symbols", lambda: qc.symbols_table(limit=10),
|
||||||
|
lambda d: f"{len(d.get('symbols') or [])} of {d.get('n_matched')} rows, "
|
||||||
|
f"facets {list((d.get('facets') or {}).keys())}")
|
||||||
|
probe("GET /prices", lambda: qc.prices(syms, start=args.start),
|
||||||
|
lambda d: f"{len(d.get('index') or [])} dates x {len(d.get('columns') or [])} symbols")
|
||||||
|
probe("GET /returns", lambda: qc.returns(syms, start=args.start),
|
||||||
|
lambda d: f"{d.get('kind')}, {len(d.get('index') or [])} dates")
|
||||||
|
probe("GET /market/index", qc.market_index,
|
||||||
|
lambda d: f"mean_ann {d.get('mean_ann'):.2%}, vol_ann {d.get('vol_ann'):.2%}")
|
||||||
|
probe("GET /market/weights", lambda: qc.market_weights(max_assets=20),
|
||||||
|
lambda d: f"{len(d.get('weights') or {})} weights, sum {sum((d.get('weights') or {}).values()):.4f}")
|
||||||
|
|
||||||
|
probe("POST /risk/assets", lambda: qc.asset_risk(syms, horizon="week"),
|
||||||
|
lambda r: f"{len(r)} rows, cvar95_hist {r[0].get('cvar95_hist'):.4f}")
|
||||||
|
probe("POST /risk/assets (drift=zero, space=value)",
|
||||||
|
lambda: qc.asset_risk(syms, horizon="week", drift="zero", space="value"),
|
||||||
|
lambda r: f"cvar95_normal_ewm {r[0].get('cvar95_normal_ewm'):.4f}")
|
||||||
|
probe("POST /risk/series", lambda: qc.risk_series(syms[0], horizon="week"),
|
||||||
|
lambda d: f"{len(d.get('index') or [])} dates")
|
||||||
|
probe("POST /risk/covariance", lambda: qc.covariance(syms),
|
||||||
|
lambda d: f"{len(d.get('columns') or [])} assets")
|
||||||
|
probe("POST /risk/correlation", lambda: qc.correlation(syms),
|
||||||
|
lambda d: f"diagonal {d['data'][0][0]:.3f}")
|
||||||
|
probe("POST /risk/portfolio", lambda: qc.portfolio_risk(holdings, cash=10_000, horizon="week"),
|
||||||
|
lambda d: f"value {d.get('value', 0):,.0f}, "
|
||||||
|
f"CVaR95 {((d.get('confidence') or {}).get('0.95') or {}).get('cvar_money', 0):,.0f}")
|
||||||
|
|
||||||
|
probe("POST /performance/assets", lambda: qc.asset_performance(syms),
|
||||||
|
lambda r: f"{len(r)} rows, sharpe {r[0].get('sharpe')}")
|
||||||
|
probe("POST /performance/rolling", lambda: qc.rolling_performance(syms[0], window=252),
|
||||||
|
lambda d: f"{len(d.get('index') or [])} dates")
|
||||||
|
probe("POST /performance/portfolio", lambda: qc.portfolio_performance(holdings),
|
||||||
|
lambda d: f"{len(d)} figures")
|
||||||
|
|
||||||
|
probe("POST /model/expected-returns",
|
||||||
|
lambda: qc.expected_returns(max_assets=args.n_symbols, symbols=args.symbols),
|
||||||
|
lambda d: f"market return {(d.get('market_return_ann') or 0):.2%}")
|
||||||
|
probe("POST /model/covariance", lambda: qc.model_covariance(syms),
|
||||||
|
lambda d: f"shrinkage {d.get('shrinkage_applied')}")
|
||||||
|
probe("POST /model/scenarios",
|
||||||
|
lambda: qc.scenarios(syms, model="gbm", n_scenarios=1024, holdings=holdings),
|
||||||
|
lambda d: f"{(d.get('description') or {}).get('n_scenarios')} scenarios, "
|
||||||
|
f"hash {(d.get('description') or {}).get('hash')}")
|
||||||
|
|
||||||
|
def optimise() -> dict:
|
||||||
|
body = {"cash": 1_000_000, "max_assets": max(len(syms), 12), "horizon": "week",
|
||||||
|
"symbols": args.symbols, "solver": args.solver, "time_limit": args.time_limit}
|
||||||
|
body.update({"scenario_model": "gbm", "n_scenarios": 4096, "cvar_limit_fraction": 0.05}
|
||||||
|
if args.full else {"scenario_model": "gaussian_exact"})
|
||||||
|
res = qc.optimize(**{k: v for k, v in body.items() if v is not None})
|
||||||
|
if res.get("status") not in ("Optimal", "FeasibleFound"):
|
||||||
|
raise AssertionError(f"status {res.get('status')}: {'; '.join(res.get('warnings') or [])}"[:120])
|
||||||
|
return res
|
||||||
|
|
||||||
|
probe(f"POST /optimize ({'gbm, 4096 scenarios' if args.full else 'gaussian_exact'})", optimise,
|
||||||
|
lambda r: f"{r.get('status')} on {r.get('solver')}, {r.get('n_positions')} positions, "
|
||||||
|
f"CVaR {r.get('cvar', 0):,.0f}")
|
||||||
|
|
||||||
|
probe("GET /jobs", qc.jobs, lambda r: f"{len(r)} jobs")
|
||||||
|
probe("GET /snapshots", qc.snapshots, lambda r: f"{len(r)} snapshots")
|
||||||
|
probe("POST /snapshots", lambda: qc.create_snapshot(f"selftest-{int(time.time())}"),
|
||||||
|
lambda d: str(d.get("path"))[:60],
|
||||||
|
skip="" if args.write else "writes to the store (--write)")
|
||||||
|
probe("POST /jobs/update", qc.start_update, lambda d: f"job {d.get('job_id')}",
|
||||||
|
skip="" if args.write else "downloads data (--write)")
|
||||||
|
probe("POST /jobs/setup", lambda: qc.start_setup(offline=True), lambda d: f"job {d.get('job_id')}",
|
||||||
|
skip="" if args.write else "rebuilds the universe (--write)")
|
||||||
|
|
||||||
|
if args.json:
|
||||||
|
return _json(results)
|
||||||
|
_records(results, round_to=0)
|
||||||
|
failed = [r for r in results if r["result"] == "FAIL"]
|
||||||
|
ok = [r for r in results if r["result"] == "ok"]
|
||||||
|
skipped = [r for r in results if r["result"] == "skipped"]
|
||||||
|
print(f"\n{len(ok)} ok, {len(failed)} failed, {len(skipped)} skipped "
|
||||||
|
f"in {sum(r['ms'] for r in results) / 1000:.1f} s against {qc.base_url}")
|
||||||
|
for r in failed:
|
||||||
|
print(f" FAIL {r['endpoint']}: {r['evidence']}")
|
||||||
|
return 1 if failed else 0
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# parser
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
|
p = argparse.ArgumentParser(prog="quantcore-api", description=__doc__,
|
||||||
|
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
p.add_argument("--url", default=None, help=f"API base URL (default: $QUANTCORE_API_URL or {default_url()})")
|
||||||
|
p.add_argument("--timeout", type=float, default=900.0, help="seconds per request")
|
||||||
|
p.add_argument("--json", action="store_true", help="print the raw JSON answer instead of a table")
|
||||||
|
sub = p.add_subparsers(dest="command", required=True)
|
||||||
|
|
||||||
|
# the same three options after the subcommand, where anyone would try them first
|
||||||
|
# ('quantcore-api optimize --json'). SUPPRESS keeps the value parsed before the subcommand
|
||||||
|
# unless the flag is repeated after it.
|
||||||
|
common = argparse.ArgumentParser(add_help=False)
|
||||||
|
common.add_argument("--url", default=argparse.SUPPRESS)
|
||||||
|
common.add_argument("--timeout", type=float, default=argparse.SUPPRESS)
|
||||||
|
common.add_argument("--json", action="store_true", default=argparse.SUPPRESS)
|
||||||
|
|
||||||
|
def add(name: str, help_: str, func: Any) -> argparse.ArgumentParser:
|
||||||
|
sp = sub.add_parser(name, help=help_, parents=[common])
|
||||||
|
sp.set_defaults(func=func)
|
||||||
|
return sp
|
||||||
|
|
||||||
|
def symbols_arg(sp: argparse.ArgumentParser) -> argparse.ArgumentParser:
|
||||||
|
sp.add_argument("symbols", nargs="+", help="Yahoo symbols, e.g. NESN.SW AAPL ASML.AS")
|
||||||
|
return sp
|
||||||
|
|
||||||
|
def window_args(sp: argparse.ArgumentParser) -> argparse.ArgumentParser:
|
||||||
|
sp.add_argument("--window", type=int, default=200, help="moving-window length in periods")
|
||||||
|
sp.add_argument("--lam", type=float, default=0.94, help="EWM decay")
|
||||||
|
return sp
|
||||||
|
|
||||||
|
add("health", "store status, cuOpt reachability, version", cmd_health)
|
||||||
|
add("endpoints", "list every endpoint from the service's own OpenAPI schema", cmd_endpoints)
|
||||||
|
|
||||||
|
up = add("universe", "the investable universe (filter by region, sector, country)", cmd_universe)
|
||||||
|
up.add_argument("--region"), up.add_argument("--sector"), up.add_argument("--country")
|
||||||
|
up.add_argument("--limit", type=int, default=25)
|
||||||
|
|
||||||
|
sy = add("symbol", "one universe row", cmd_symbol)
|
||||||
|
sy.add_argument("symbol")
|
||||||
|
|
||||||
|
for name, func, what in (("prices", cmd_prices, "adjusted closes"),
|
||||||
|
("returns", cmd_returns, "log-returns")):
|
||||||
|
sp = symbols_arg(add(name, what, func))
|
||||||
|
sp.add_argument("--freq", default="daily", choices=["daily", "weekly", "monthly"])
|
||||||
|
sp.add_argument("--currency", default="base", choices=["base", "local"])
|
||||||
|
sp.add_argument("--start"), sp.add_argument("--end")
|
||||||
|
sp.add_argument("--tail", type=int, default=10, help="print only the last N rows (0: all)")
|
||||||
|
|
||||||
|
mk = add("market", "the cap-weighted broad market: 'index' or 'weights'", cmd_market)
|
||||||
|
mk.add_argument("what", nargs="?", default="index", choices=["index", "weights"])
|
||||||
|
mk.add_argument("--freq", default="weekly", choices=["daily", "weekly", "monthly"])
|
||||||
|
mk.add_argument("--start"), mk.add_argument("--end")
|
||||||
|
mk.add_argument("--max-assets", type=int, default=None), mk.add_argument("--limit", type=int, default=25)
|
||||||
|
|
||||||
|
rk = window_args(symbols_arg(add("risk", "per-stock volatility, VaR and CVaR", cmd_risk)))
|
||||||
|
rk.add_argument("--horizon", default="week", help="'day' | 'week' | 'month' | 'quarter' | 'year' | periods")
|
||||||
|
rk.add_argument("--freq", default="daily", choices=["daily", "weekly", "monthly"])
|
||||||
|
rk.add_argument("--currency", default="base", choices=["base", "local"])
|
||||||
|
rk.add_argument("--drift", default="estimated", choices=["estimated", "zero"],
|
||||||
|
help="'zero': volatility only, the market-risk convention")
|
||||||
|
rk.add_argument("--space", default="log", choices=["log", "value"],
|
||||||
|
help="'value': as a fraction of the position's value, which cannot exceed 100 %%")
|
||||||
|
rk.add_argument("--confidence", type=float, nargs="+", default=[0.95, 0.99])
|
||||||
|
|
||||||
|
rp = add("risk-portfolio", "VaR/CVaR of a portfolio held in share numbers", cmd_risk_portfolio)
|
||||||
|
rp.add_argument("--holdings", required=True, help='JSON: {"NESN.SW": 500, "AAPL": 100}')
|
||||||
|
rp.add_argument("--cash", type=float, default=0.0)
|
||||||
|
rp.add_argument("--horizon", default="week")
|
||||||
|
rp.add_argument("--method", default="normal", choices=["normal", "historical"])
|
||||||
|
rp.add_argument("--confidence", type=float, nargs="+", default=[0.95, 0.99])
|
||||||
|
|
||||||
|
for name in ("correlation", "covariance"):
|
||||||
|
sp = window_args(symbols_arg(add(name, f"{name} matrix as measured", cmd_matrix)))
|
||||||
|
sp.add_argument("--method", default="ewm", choices=["rolling", "ewm"])
|
||||||
|
sp.add_argument("--freq", default="daily", choices=["daily", "weekly", "monthly"])
|
||||||
|
|
||||||
|
pf = symbols_arg(add("performance", "annualised return, Sharpe, drawdown, skew, kurtosis", cmd_performance))
|
||||||
|
pf.add_argument("--freq", default="weekly", choices=["daily", "weekly", "monthly"])
|
||||||
|
pf.add_argument("--currency", default="base", choices=["base", "local"])
|
||||||
|
pf.add_argument("--start")
|
||||||
|
|
||||||
|
er = add("expected-returns", "Black-Litterman equilibrium and posterior returns", cmd_expected_returns)
|
||||||
|
er.add_argument("--symbols", nargs="*", default=None)
|
||||||
|
er.add_argument("--max-assets", type=int, default=100)
|
||||||
|
er.add_argument("--tau", type=float, default=0.05)
|
||||||
|
er.add_argument("--years", type=int, default=10)
|
||||||
|
er.add_argument("--omega-method", default="idzorek", choices=["idzorek", "he_litterman"])
|
||||||
|
er.add_argument("--view", action="append", metavar="SYM:SENTIMENT[:CONF[:STRENGTH]]",
|
||||||
|
help="a sentiment view; repeatable")
|
||||||
|
er.add_argument("--limit", type=int, default=25)
|
||||||
|
|
||||||
|
mc = window_args(symbols_arg(add("model-covariance", "the covariance the models use, with its "
|
||||||
|
"conditioning", cmd_model_covariance)))
|
||||||
|
mc.add_argument("--cov-method", default="ewm", choices=["rolling", "ewm"])
|
||||||
|
mc.add_argument("--shrinkage", default="auto")
|
||||||
|
mc.add_argument("--freq", default="daily", choices=["daily", "weekly"])
|
||||||
|
mc.add_argument("--horizon", default="week")
|
||||||
|
|
||||||
|
sc = window_args(symbols_arg(add("scenarios", "a scenario set and its statistics", cmd_scenarios)))
|
||||||
|
sc.add_argument("--model", default="gbm", choices=["gbm", "empirical"])
|
||||||
|
sc.add_argument("--n-scenarios", type=int, default=4096)
|
||||||
|
sc.add_argument("--horizon", default="week")
|
||||||
|
sc.add_argument("--freq", default="daily", choices=["daily", "weekly"])
|
||||||
|
sc.add_argument("--seed", type=int, default=42)
|
||||||
|
sc.add_argument("--sampling", default="sobol", choices=["sobol", "mc", "antithetic", "lhs", "balanced"])
|
||||||
|
sc.add_argument("--cov-method", default="ewm", choices=["rolling", "ewm"])
|
||||||
|
sc.add_argument("--shrinkage", default="auto")
|
||||||
|
sc.add_argument("--expected-returns", default="zero", choices=["zero", "historical", "black_litterman"])
|
||||||
|
sc.add_argument("--importance-sampling", default="auto", choices=["auto", "market", "adaptive", "off"])
|
||||||
|
sc.add_argument("--holdings", help='JSON: {"NESN.SW": 500}; else the set is reported per stock')
|
||||||
|
sc.add_argument("--value", type=float, default=1_000_000.0)
|
||||||
|
sc.add_argument("--confidence", type=float, nargs="+", default=[0.95, 0.99])
|
||||||
|
|
||||||
|
op = add("optimize", "mean-CVaR MILP in integer shares", cmd_optimize)
|
||||||
|
op.add_argument("--request", help="JSON file with the full request; its fields win over the flags")
|
||||||
|
op.add_argument("--cash", type=float, default=1_000_000.0)
|
||||||
|
op.add_argument("--holdings", help='JSON dict -> rebalancing mode')
|
||||||
|
op.add_argument("--symbols", nargs="*", default=None, help="candidates; default: largest by market cap")
|
||||||
|
op.add_argument("--max-assets", type=int, default=100)
|
||||||
|
op.add_argument("--horizon", default="week")
|
||||||
|
op.add_argument("--freq", default="daily", choices=["daily", "weekly"])
|
||||||
|
op.add_argument("--confidence", type=float, default=0.95)
|
||||||
|
op.add_argument("--cvar-limit", type=float, default=None, help="max CVaR in CHF over the horizon")
|
||||||
|
op.add_argument("--objective", default="max_return", choices=["max_return", "min_cvar", "utility"])
|
||||||
|
op.add_argument("--scenario-model", default="empirical", choices=["gbm", "empirical", "gaussian_exact"])
|
||||||
|
op.add_argument("--n-scenarios", type=int, default=None,
|
||||||
|
help="default: 16 per candidate within [4096, 16384], rounded up to a power of two")
|
||||||
|
op.add_argument("--seed", type=int, default=42)
|
||||||
|
op.add_argument("--expected-returns", default="black_litterman",
|
||||||
|
choices=["black_litterman", "historical", "zero"])
|
||||||
|
op.add_argument("--risk-drift", default="model", choices=["model", "zero"],
|
||||||
|
help="'zero': measure the CVaR on centred scenarios, so a bullish forecast "
|
||||||
|
"cannot relax the risk limit")
|
||||||
|
op.add_argument("--importance-sampling", default="auto", choices=["auto", "market", "adaptive", "off"])
|
||||||
|
op.add_argument("--row-generation", default="auto", choices=["auto", "on", "off"])
|
||||||
|
op.add_argument("--max-weight", type=float, default=None, help="cap per stock (default 0.10)")
|
||||||
|
op.add_argument("--max-sector-weight", type=float, default=None, help="cap per GICS sector (default 0.25)")
|
||||||
|
op.add_argument("--view", action="append", metavar="SYM:SENTIMENT[:CONF[:STRENGTH]]", help="repeatable")
|
||||||
|
op.add_argument("--solver", default="auto", choices=["auto", "cuopt", "highs"])
|
||||||
|
op.add_argument("--time-limit", type=float, default=300.0, help="seconds for the whole request")
|
||||||
|
op.add_argument("--quiet", action="store_true", help="summary only, no position table")
|
||||||
|
|
||||||
|
jb = add("jobs", "list background jobs, or one with --job-id", cmd_jobs)
|
||||||
|
jb.add_argument("--job-id", default=None)
|
||||||
|
|
||||||
|
st = add("setup", "start the universe/history build (background job)", cmd_setup)
|
||||||
|
st.add_argument("--offline", action="store_true", help="use the candidate lists shipped in the package")
|
||||||
|
st.add_argument("--max-per-region", type=int, default=None)
|
||||||
|
st.add_argument("--wait", action="store_true"), st.add_argument("--poll", type=float, default=10.0)
|
||||||
|
|
||||||
|
ud = add("update", "start the price/FX refresh (background job)", cmd_update)
|
||||||
|
ud.add_argument("--full", action="store_true")
|
||||||
|
ud.add_argument("--wait", action="store_true"), ud.add_argument("--poll", type=float, default=5.0)
|
||||||
|
|
||||||
|
sn = add("snapshots", "list snapshots, or --create one", cmd_snapshots)
|
||||||
|
sn.add_argument("--create", action="store_true"), sn.add_argument("--name", default=None)
|
||||||
|
|
||||||
|
se = add("selftest", "call every endpoint and report what it answered", cmd_selftest)
|
||||||
|
se.add_argument("--full", action="store_true", help="optimise on a real scenario set, not gaussian_exact")
|
||||||
|
se.add_argument("--write", action="store_true", help="also probe the endpoints that write or download")
|
||||||
|
se.add_argument("--symbols", nargs="*", default=None, help="default: the largest few in the universe")
|
||||||
|
se.add_argument("--n-symbols", type=int, default=3)
|
||||||
|
se.add_argument("--start", default=None, help="first date for the price/return probes")
|
||||||
|
se.add_argument("--solver", default="auto", choices=["auto", "cuopt", "highs"])
|
||||||
|
se.add_argument("--time-limit", type=float, default=120.0)
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
args = build_parser().parse_args(argv)
|
||||||
|
try:
|
||||||
|
return args.func(args)
|
||||||
|
except QuantcoreError as exc:
|
||||||
|
print(f"error: {exc}", file=sys.stderr)
|
||||||
|
if exc.unreachable:
|
||||||
|
print("is the service running? start it with 'quantcore serve', or pass --url", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
print("interrupted", file=sys.stderr)
|
||||||
|
return 130
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
253
quantcore/cli.py
Normal file
253
quantcore/cli.py
Normal file
@ -0,0 +1,253 @@
|
|||||||
|
"""Command-line interface: ``quantcore <command>``.
|
||||||
|
|
||||||
|
quantcore setup [--offline] [--max-per-region N] build the universe once (candidates,
|
||||||
|
metadata, full price + FX history)
|
||||||
|
quantcore reselect re-run only the selection step
|
||||||
|
quantcore clean re-apply price cleaning to stored data
|
||||||
|
quantcore update [--full] fetch the newest prices / FX
|
||||||
|
quantcore snapshot [--name NAME] freeze the data for reproducible demos
|
||||||
|
quantcore status what the store holds, cuOpt health
|
||||||
|
quantcore risk SYMBOL [SYMBOL ...] [--horizon week] [--freq daily] [--window 200] [--lam 0.94]
|
||||||
|
quantcore performance SYMBOL [SYMBOL ...] [--freq weekly]
|
||||||
|
quantcore optimize [--cash 1e6] [--cvar-limit 50000] [--confidence 0.95] [--horizon week] ...
|
||||||
|
quantcore serve [--host 127.0.0.1] [--port 5001] run the HTTP API (+ the test bench at /ui/)
|
||||||
|
|
||||||
|
These commands run the services in this process and need the data directory locally. To drive a
|
||||||
|
*running* service over HTTP instead -- from any machine, without the data -- use ``quantcore-api``
|
||||||
|
(:mod:`quantcore.apicli`), whose ``selftest`` also checks a deployment endpoint by endpoint.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from quantcore.config import Settings, get_settings
|
||||||
|
|
||||||
|
log = logging.getLogger("quantcore.cli")
|
||||||
|
|
||||||
|
|
||||||
|
def _settings(args) -> Settings:
|
||||||
|
s = get_settings()
|
||||||
|
if getattr(args, "data_dir", None):
|
||||||
|
s = s.model_copy(update={"data_dir": args.data_dir})
|
||||||
|
if getattr(args, "cuopt_url", None):
|
||||||
|
s = s.model_copy(update={"cuopt_url": args.cuopt_url})
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_setup(args) -> int:
|
||||||
|
from quantcore.universe.setup_service import SetupService
|
||||||
|
|
||||||
|
report = SetupService(_settings(args)).run(allow_web=not args.offline, max_per_region=args.max_per_region)
|
||||||
|
print(json.dumps(report.as_dict(), indent=2, default=str))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_reselect(args) -> int:
|
||||||
|
from quantcore.universe.setup_service import SetupService
|
||||||
|
|
||||||
|
report = SetupService(_settings(args)).rebuild_selection()
|
||||||
|
print(json.dumps(report.as_dict(), indent=2, default=str))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_update(args) -> int:
|
||||||
|
from quantcore.data.download_service import DataDownloadService
|
||||||
|
|
||||||
|
report = DataDownloadService(_settings(args)).update(full=args.full)
|
||||||
|
print(json.dumps(report.as_dict(), indent=2, default=str))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_clean(args) -> int:
|
||||||
|
from quantcore.data.download_service import DataDownloadService
|
||||||
|
|
||||||
|
report = DataDownloadService(_settings(args)).clean()
|
||||||
|
print(json.dumps(report.as_dict(), indent=2, default=str))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_snapshot(args) -> int:
|
||||||
|
from quantcore.data.store import DataStore
|
||||||
|
|
||||||
|
path = DataStore(_settings(args)).create_snapshot(args.name)
|
||||||
|
print(path)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_status(args) -> int:
|
||||||
|
from quantcore.data.store import DataStore
|
||||||
|
from quantcore.optimization.solvers import CuOptSolver
|
||||||
|
|
||||||
|
s = _settings(args)
|
||||||
|
out = {"store": DataStore(s).status(), "cuopt": CuOptSolver(s).health(), "cuopt_url": s.cuopt_url}
|
||||||
|
print(json.dumps(out, indent=2, default=str))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_risk(args) -> int:
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.measurement.risk import RiskService
|
||||||
|
|
||||||
|
md = MarketDataService(_settings(args), snapshot=args.snapshot)
|
||||||
|
table = RiskService(md).asset_risk(args.symbols, freq=args.freq, window=args.window, lam=args.lam,
|
||||||
|
horizon=args.horizon, currency=args.currency)
|
||||||
|
with pd.option_context("display.width", 200, "display.max_columns", 30):
|
||||||
|
print(table.round(5).T)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_performance(args) -> int:
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.measurement.performance import PerformanceService
|
||||||
|
|
||||||
|
s = _settings(args)
|
||||||
|
md = MarketDataService(s, snapshot=args.snapshot)
|
||||||
|
table = PerformanceService(md, s.risk_free_rate).asset_performance(args.symbols, freq=args.freq,
|
||||||
|
currency=args.currency, start=args.start)
|
||||||
|
with pd.option_context("display.width", 200, "display.max_columns", 30):
|
||||||
|
print(table.round(4).T)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_optimize(args) -> int:
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.optimization.service import OptimizationRequest, OptimizationService
|
||||||
|
|
||||||
|
s = _settings(args)
|
||||||
|
md = MarketDataService(s, snapshot=args.snapshot)
|
||||||
|
kwargs = {}
|
||||||
|
if args.request:
|
||||||
|
kwargs = json.loads(open(args.request).read())
|
||||||
|
kwargs.setdefault("cash", args.cash)
|
||||||
|
kwargs.setdefault("horizon", args.horizon)
|
||||||
|
kwargs.setdefault("confidence", args.confidence)
|
||||||
|
if args.cvar_limit is not None:
|
||||||
|
kwargs["cvar_limit"] = args.cvar_limit
|
||||||
|
if args.symbols:
|
||||||
|
kwargs["symbols"] = args.symbols
|
||||||
|
kwargs.setdefault("max_assets", args.max_assets)
|
||||||
|
kwargs.setdefault("scenario_model", args.scenario_model)
|
||||||
|
if args.n_scenarios is not None:
|
||||||
|
kwargs.setdefault("n_scenarios", args.n_scenarios)
|
||||||
|
kwargs.setdefault("sampling", args.sampling)
|
||||||
|
kwargs.setdefault("importance_sampling", args.importance_sampling)
|
||||||
|
kwargs.setdefault("row_generation", args.row_generation)
|
||||||
|
kwargs.setdefault("solver", args.solver)
|
||||||
|
if args.holdings:
|
||||||
|
kwargs["holdings"] = json.loads(args.holdings)
|
||||||
|
kwargs["mode"] = "rebalance"
|
||||||
|
req = OptimizationRequest(**kwargs)
|
||||||
|
res = OptimizationService(s, md).optimize(req)
|
||||||
|
if args.json:
|
||||||
|
print(json.dumps(res.as_dict(), indent=2, default=str))
|
||||||
|
else:
|
||||||
|
print(f"status={res.status} solver={res.solver} reproducible={res.reproducibility} scenarios={res.scenario_hash} "
|
||||||
|
f"key={res.scenario_key['id'] if res.scenario_key else None} "
|
||||||
|
f"wealth={res.wealth:,.0f} invested={res.invested:,.0f} cash_end={res.cash_end:,.0f} cost={res.total_cost:,.0f}")
|
||||||
|
print(f"E[P&L]={res.expected_pnl:,.0f} ({res.expected_return:.3%}) VaR={res.var:,.0f} CVaR={res.cvar:,.0f} "
|
||||||
|
f"limit={res.cvar_limit} @ {res.confidence:.0%} over {res.horizon} ({res.horizon_periods} periods)")
|
||||||
|
if res.positions:
|
||||||
|
df = pd.DataFrame(res.positions)[["symbol", "name", "sector", "price", "shares", "trade", "value", "weight"]]
|
||||||
|
with pd.option_context("display.width", 200, "display.max_rows", 500):
|
||||||
|
print(df.round(3).to_string(index=False))
|
||||||
|
print("sectors:", {k: round(v, 3) for k, v in res.sector_exposure.items()})
|
||||||
|
print("timings:", res.timings, "warnings:", res.warnings)
|
||||||
|
return 0 if res.status in ("Optimal", "FeasibleFound") else 2
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_serve(args) -> int:
|
||||||
|
import uvicorn
|
||||||
|
|
||||||
|
s = _settings(args)
|
||||||
|
uvicorn.run("quantcore.api.app:app", host=args.host or s.api_host, port=args.port or s.api_port, reload=False)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
|
p = argparse.ArgumentParser(prog="quantcore", description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
p.add_argument("--data-dir", help="override QUANTCORE_DATA_DIR")
|
||||||
|
p.add_argument("--cuopt-url", help="override QUANTCORE_CUOPT_URL")
|
||||||
|
sub = p.add_subparsers(dest="command", required=True)
|
||||||
|
|
||||||
|
sp = sub.add_parser("setup", help="build the universe and download all history (run once)")
|
||||||
|
sp.add_argument("--offline", action="store_true", help="use the static candidate lists shipped with the package")
|
||||||
|
sp.add_argument("--max-per-region", type=int, default=None, help="limit candidates per region (testing)")
|
||||||
|
sp.set_defaults(func=cmd_setup)
|
||||||
|
|
||||||
|
sub.add_parser("reselect", help="re-run only the universe selection on stored data").set_defaults(func=cmd_reselect)
|
||||||
|
|
||||||
|
up = sub.add_parser("update", help="fetch the newest prices and FX rates")
|
||||||
|
up.add_argument("--full", action="store_true", help="re-download the complete history")
|
||||||
|
up.set_defaults(func=cmd_update)
|
||||||
|
|
||||||
|
sub.add_parser("clean", help="re-apply the price cleaning to the stored history").set_defaults(func=cmd_clean)
|
||||||
|
|
||||||
|
sn = sub.add_parser("snapshot", help="freeze universe/prices/fx under snapshots/<name>")
|
||||||
|
sn.add_argument("--name", default=None)
|
||||||
|
sn.set_defaults(func=cmd_snapshot)
|
||||||
|
|
||||||
|
sub.add_parser("status", help="store status and cuOpt health").set_defaults(func=cmd_status)
|
||||||
|
|
||||||
|
rk = sub.add_parser("risk", help="per-asset volatility, VaR and CVaR")
|
||||||
|
rk.add_argument("symbols", nargs="+")
|
||||||
|
rk.add_argument("--freq", default="daily", choices=["daily", "weekly", "monthly"])
|
||||||
|
rk.add_argument("--horizon", default="day")
|
||||||
|
rk.add_argument("--window", type=int, default=200)
|
||||||
|
rk.add_argument("--lam", type=float, default=0.94)
|
||||||
|
rk.add_argument("--currency", default="base", choices=["base", "local"])
|
||||||
|
rk.add_argument("--snapshot", default=None)
|
||||||
|
rk.set_defaults(func=cmd_risk)
|
||||||
|
|
||||||
|
pf = sub.add_parser("performance", help="per-asset performance statistics")
|
||||||
|
pf.add_argument("symbols", nargs="+")
|
||||||
|
pf.add_argument("--freq", default="daily", choices=["daily", "weekly", "monthly"])
|
||||||
|
pf.add_argument("--currency", default="base", choices=["base", "local"])
|
||||||
|
pf.add_argument("--start", default=None)
|
||||||
|
pf.add_argument("--snapshot", default=None)
|
||||||
|
pf.set_defaults(func=cmd_performance)
|
||||||
|
|
||||||
|
op = sub.add_parser("optimize", help="mean-CVaR MILP optimisation")
|
||||||
|
op.add_argument("--request", help="JSON file with an OptimizationRequest (fields override the flags)")
|
||||||
|
op.add_argument("--cash", type=float, default=1_000_000.0)
|
||||||
|
op.add_argument("--holdings", help='JSON dict {"NESN.SW": 100, ...} -> rebalancing mode')
|
||||||
|
op.add_argument("--symbols", nargs="*", default=None)
|
||||||
|
op.add_argument("--max-assets", type=int, default=100)
|
||||||
|
op.add_argument("--horizon", default="week")
|
||||||
|
op.add_argument("--confidence", type=float, default=0.95)
|
||||||
|
op.add_argument("--cvar-limit", type=float, default=None, help="max CVaR in CHF over the horizon")
|
||||||
|
op.add_argument("--scenario-model", default="empirical", choices=["gbm", "empirical", "gaussian_exact"])
|
||||||
|
op.add_argument("--n-scenarios", type=int, default=None,
|
||||||
|
help="default: 16 per candidate within [4096, 16384]; rounded up to a power of two")
|
||||||
|
op.add_argument("--sampling", default="sobol", choices=["sobol", "mc", "antithetic", "lhs", "balanced"])
|
||||||
|
from quantcore.optimization.service import IMPORTANCE_SAMPLING_MODES # lazy, like every command's
|
||||||
|
|
||||||
|
op.add_argument("--importance-sampling", default="auto",
|
||||||
|
choices=list(IMPORTANCE_SAMPLING_MODES),
|
||||||
|
help="'market': tilt towards the tail of the cap-weighted portfolio; "
|
||||||
|
"'adaptive': towards a first-pass solution")
|
||||||
|
op.add_argument("--row-generation", default="auto", choices=["auto", "on", "off"])
|
||||||
|
op.add_argument("--solver", default="auto", choices=["auto", "cuopt", "highs"])
|
||||||
|
op.add_argument("--snapshot", default=None)
|
||||||
|
op.add_argument("--json", action="store_true", help="print the full result as JSON")
|
||||||
|
op.set_defaults(func=cmd_optimize)
|
||||||
|
|
||||||
|
sv = sub.add_parser("serve", help="run the HTTP API and the browser test bench (/ui/)")
|
||||||
|
sv.add_argument("--host", default=None)
|
||||||
|
sv.add_argument("--port", type=int, default=None)
|
||||||
|
sv.set_defaults(func=cmd_serve)
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
args = build_parser().parse_args(argv)
|
||||||
|
return args.func(args)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
314
quantcore/client.py
Normal file
314
quantcore/client.py
Normal file
@ -0,0 +1,314 @@
|
|||||||
|
"""HTTP client of the quant-core API -- the package seen from the outside.
|
||||||
|
|
||||||
|
The services of :mod:`quantcore` can be used in two ways. In-process, by importing
|
||||||
|
``MarketDataService``, ``RiskService``, ``OptimizationService`` and friends, which needs the data
|
||||||
|
directory on the same machine. Or over HTTP, against a running ``quantcore serve`` (or the container
|
||||||
|
on the DGX Spark), which needs nothing but a URL -- this module.
|
||||||
|
|
||||||
|
:class:`QuantcoreClient` has one method per endpoint, passes the request bodies through unchanged
|
||||||
|
(the field names are exactly those of ``quantcore.api.schemas``, so the interactive docs at
|
||||||
|
``/docs`` are its reference) and returns the parsed JSON. It adds no logic and no defaults of its
|
||||||
|
own beyond the ones the API already applies: what comes back is what the services computed.
|
||||||
|
|
||||||
|
from quantcore.client import QuantcoreClient
|
||||||
|
|
||||||
|
qc = QuantcoreClient("http://spark-6d4a:5001")
|
||||||
|
qc.health()["store"]["last_price_date"]
|
||||||
|
qc.asset_risk(["NESN.SW", "AAPL"], horizon="week", drift="zero")
|
||||||
|
res = qc.optimize(cash=1_000_000, cvar_limit=40_000, horizon="week", risk_drift="zero")
|
||||||
|
res["status"], res["cvar"], len(res["positions"])
|
||||||
|
|
||||||
|
Errors: any answer that is not 2xx raises :class:`QuantcoreError` carrying the status code and the
|
||||||
|
API's ``detail`` message; an unreachable server raises it too, naming the URL that was tried. The
|
||||||
|
methods that start background work (``start_setup``, ``start_update``) return immediately with a job
|
||||||
|
id -- :meth:`QuantcoreClient.wait_for_job` polls it to completion.
|
||||||
|
|
||||||
|
The transport is injectable (``session=``): anything with a ``request(method, url, params=,
|
||||||
|
json=, timeout=)`` method that returns an object with ``status_code``, ``json()`` and ``text``.
|
||||||
|
``requests.Session`` is the default; ``fastapi.testclient.TestClient`` satisfies it too, which is
|
||||||
|
how the tests exercise this client against the app in-process.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from typing import Any, Self
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
|
DEFAULT_URL = "http://127.0.0.1:5001"
|
||||||
|
DEFAULT_TIMEOUT = 900.0 # an optimisation may legitimately run for minutes
|
||||||
|
JOB_STATES_DONE = ("done", "failed")
|
||||||
|
|
||||||
|
|
||||||
|
def default_url() -> str:
|
||||||
|
"""``QUANTCORE_API_URL`` if set, else ``http://127.0.0.1:5001``."""
|
||||||
|
return os.environ.get("QUANTCORE_API_URL", DEFAULT_URL)
|
||||||
|
|
||||||
|
|
||||||
|
class QuantcoreError(RuntimeError):
|
||||||
|
"""An API answer that is not 2xx, or a server that could not be reached.
|
||||||
|
|
||||||
|
``status_code`` is ``None`` when the request never got an answer (connection refused, timeout,
|
||||||
|
DNS failure) -- distinguishing "the service said no" from "there is no service there".
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, message: str, *, status_code: int | None = None, method: str = "", path: str = ""):
|
||||||
|
super().__init__(message)
|
||||||
|
self.status_code = status_code
|
||||||
|
self.method = method
|
||||||
|
self.path = path
|
||||||
|
|
||||||
|
@property
|
||||||
|
def unreachable(self) -> bool:
|
||||||
|
return self.status_code is None
|
||||||
|
|
||||||
|
|
||||||
|
def _retrying_session() -> requests.Session:
|
||||||
|
"""A session that retries an idempotent request (GET/HEAD/OPTIONS) when the connection drops
|
||||||
|
before the answer arrives -- typically a kept-alive connection that the server closed between two
|
||||||
|
calls: uvicorn closes idle connections after 5 s, which is also the default job poll interval.
|
||||||
|
POSTs are never retried after they were sent: an optimisation or a job must not run twice."""
|
||||||
|
from requests.adapters import HTTPAdapter
|
||||||
|
from urllib3.util.retry import Retry
|
||||||
|
|
||||||
|
retry = Retry(total=2, connect=2, read=2, status=0, redirect=0, other=0, backoff_factor=0.2,
|
||||||
|
allowed_methods=frozenset({"GET", "HEAD", "OPTIONS"}), raise_on_status=False)
|
||||||
|
session = requests.Session()
|
||||||
|
adapter = HTTPAdapter(max_retries=retry)
|
||||||
|
session.mount("http://", adapter)
|
||||||
|
session.mount("https://", adapter)
|
||||||
|
return session
|
||||||
|
|
||||||
|
|
||||||
|
class QuantcoreClient:
|
||||||
|
"""Client of one quant-core API instance.
|
||||||
|
|
||||||
|
``base_url``: e.g. ``http://127.0.0.1:5001`` or ``http://spark-6d4a:5001``; a trailing slash and
|
||||||
|
a trailing ``/ui`` are stripped, so a URL copied out of the browser works.
|
||||||
|
``timeout``: seconds per request, overridable per call (``timeout=`` on any method).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, base_url: str | None = None, timeout: float = DEFAULT_TIMEOUT, session: Any = None):
|
||||||
|
self.base_url = (base_url or default_url()).rstrip("/").removesuffix("/ui")
|
||||||
|
self.timeout = float(timeout)
|
||||||
|
self._session = session
|
||||||
|
self._owns_session = session is None
|
||||||
|
|
||||||
|
# -- transport ------------------------------------------------------------------------------
|
||||||
|
@property
|
||||||
|
def session(self) -> Any:
|
||||||
|
if self._session is None:
|
||||||
|
self._session = _retrying_session()
|
||||||
|
return self._session
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
if self._owns_session and self._session is not None:
|
||||||
|
self._session.close()
|
||||||
|
self._session = None
|
||||||
|
|
||||||
|
def __enter__(self) -> Self:
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *exc: object) -> None:
|
||||||
|
self.close()
|
||||||
|
|
||||||
|
def request(self, method: str, path: str, *, params: dict | None = None, json: Any = None,
|
||||||
|
timeout: float | None = None) -> Any:
|
||||||
|
"""One call against the API. ``params``/``json`` entries that are ``None`` are dropped, so a
|
||||||
|
method can pass every optional argument and let the API apply its own defaults."""
|
||||||
|
if params:
|
||||||
|
params = {k: v for k, v in params.items() if v is not None}
|
||||||
|
if isinstance(json, dict):
|
||||||
|
json = {k: v for k, v in json.items() if v is not None}
|
||||||
|
url = f"{self.base_url}{path}"
|
||||||
|
try:
|
||||||
|
res = self.session.request(method, url, params=params, json=json,
|
||||||
|
timeout=timeout if timeout is not None else self.timeout)
|
||||||
|
except requests.RequestException as exc:
|
||||||
|
raise QuantcoreError(f"cannot reach the quant-core API at {url}: {exc}",
|
||||||
|
method=method, path=path) from exc
|
||||||
|
if not 200 <= res.status_code < 300:
|
||||||
|
raise QuantcoreError(f"{method} {path} -> {res.status_code}: {_detail(res)}",
|
||||||
|
status_code=res.status_code, method=method, path=path)
|
||||||
|
return res.json()
|
||||||
|
|
||||||
|
def get(self, path: str, **kw: Any) -> Any:
|
||||||
|
return self.request("GET", path, **kw)
|
||||||
|
|
||||||
|
def post(self, path: str, **kw: Any) -> Any:
|
||||||
|
return self.request("POST", path, **kw)
|
||||||
|
|
||||||
|
# -- system ---------------------------------------------------------------------------------
|
||||||
|
def health(self, **kw: Any) -> dict:
|
||||||
|
"""Store status (symbols, days, last price date), cuOpt reachability, base currency."""
|
||||||
|
return self.get("/health", **kw)
|
||||||
|
|
||||||
|
def openapi(self, **kw: Any) -> dict:
|
||||||
|
"""The OpenAPI schema -- the machine-readable list of every endpoint and field."""
|
||||||
|
return self.get("/openapi.json", **kw)
|
||||||
|
|
||||||
|
# -- data -----------------------------------------------------------------------------------
|
||||||
|
def universe(self, region: str | None = None, sector: str | None = None, country: str | None = None,
|
||||||
|
**kw: Any) -> list[dict]:
|
||||||
|
"""The investable universe, one row per stock (metadata, market cap, coverage)."""
|
||||||
|
return self.get("/universe", params={"region": region, "sector": sector, "country": country}, **kw)
|
||||||
|
|
||||||
|
def symbol(self, symbol: str, **kw: Any) -> dict:
|
||||||
|
"""One universe row."""
|
||||||
|
return self.get(f"/universe/{symbol}", **kw)
|
||||||
|
|
||||||
|
def symbols_table(self, region: str | None = None, sector: str | None = None, country: str | None = None,
|
||||||
|
search: str | None = None, limit: int = 200, sort: str = "market_cap", **kw: Any) -> dict:
|
||||||
|
"""The universe as a table plus the filter facets (counts per region, sector, country) --
|
||||||
|
one call for a stock picker."""
|
||||||
|
return self.get("/data/symbols", params={"region": region, "sector": sector, "country": country,
|
||||||
|
"search": search, "limit": limit, "sort": sort}, **kw)
|
||||||
|
|
||||||
|
def prices(self, symbols: list[str], start: str | None = None, end: str | None = None,
|
||||||
|
currency: str = "base", freq: str = "daily", **kw: Any) -> dict:
|
||||||
|
"""Adjusted closes as ``{"index": [...], "columns": [...], "data": [[...]]}``."""
|
||||||
|
return self.get("/prices", params={"symbols": ",".join(symbols), "start": start, "end": end,
|
||||||
|
"currency": currency, "freq": freq}, **kw)
|
||||||
|
|
||||||
|
def returns(self, symbols: list[str], start: str | None = None, end: str | None = None,
|
||||||
|
currency: str = "base", freq: str = "daily", **kw: Any) -> dict:
|
||||||
|
"""Log-returns in the same frame layout as :meth:`prices`."""
|
||||||
|
return self.get("/returns", params={"symbols": ",".join(symbols), "start": start, "end": end,
|
||||||
|
"currency": currency, "freq": freq}, **kw)
|
||||||
|
|
||||||
|
def market_index(self, start: str | None = None, end: str | None = None, freq: str = "weekly",
|
||||||
|
**kw: Any) -> dict:
|
||||||
|
"""The cap-weighted broad-market index: its return series and annualised mean/volatility."""
|
||||||
|
return self.get("/market/index", params={"start": start, "end": end, "freq": freq}, **kw)
|
||||||
|
|
||||||
|
def market_weights(self, max_assets: int | None = None, **kw: Any) -> dict:
|
||||||
|
"""Current market-capitalisation weights (the Black-Litterman equilibrium portfolio)."""
|
||||||
|
return self.get("/market/weights", params={"max_assets": max_assets}, **kw)
|
||||||
|
|
||||||
|
# -- risk -----------------------------------------------------------------------------------
|
||||||
|
def asset_risk(self, symbols: list[str], **body: Any) -> list[dict]:
|
||||||
|
"""Per-stock volatility, VaR and CVaR (normal and historical, rolling and EWM).
|
||||||
|
|
||||||
|
``horizon``, ``freq``, ``window``, ``lam``, ``confidence``, ``currency``, and:
|
||||||
|
``drift="zero"`` leaves the estimated mean out of the parametric figures (the market-risk
|
||||||
|
convention, and the only sane choice at long horizons), ``space="value"`` reports them as a
|
||||||
|
fraction of the position's value instead of as log-returns.
|
||||||
|
"""
|
||||||
|
return self.post("/risk/assets", json={"symbols": symbols, **body})
|
||||||
|
|
||||||
|
def risk_series(self, symbol: str, **body: Any) -> dict:
|
||||||
|
"""Rolling and EWM volatility / VaR / CVaR through time for one stock."""
|
||||||
|
return self.post("/risk/series", json={"symbol": symbol, **body})
|
||||||
|
|
||||||
|
def covariance(self, symbols: list[str], **body: Any) -> dict:
|
||||||
|
"""Covariance matrix as measured (``method="rolling"|"ewm"``, ``window``, ``lam``,
|
||||||
|
``shrinkage``). For the estimator the *models* use, see :meth:`model_covariance`."""
|
||||||
|
return self.post("/risk/covariance", json={"symbols": symbols, **body})
|
||||||
|
|
||||||
|
def correlation(self, symbols: list[str], **body: Any) -> dict:
|
||||||
|
"""Correlation matrix, same parameters as :meth:`covariance`."""
|
||||||
|
return self.post("/risk/correlation", json={"symbols": symbols, **body})
|
||||||
|
|
||||||
|
def portfolio_risk(self, holdings: dict[str, float], **body: Any) -> dict:
|
||||||
|
"""VaR/CVaR of a portfolio held in *share numbers* (plus ``cash``), normal or historical."""
|
||||||
|
return self.post("/risk/portfolio", json={"holdings": holdings, **body})
|
||||||
|
|
||||||
|
# -- performance ----------------------------------------------------------------------------
|
||||||
|
def asset_performance(self, symbols: list[str], **body: Any) -> list[dict]:
|
||||||
|
"""Annualised return, volatility, Sharpe, Sortino, drawdown, Calmar, skew, kurtosis."""
|
||||||
|
return self.post("/performance/assets", json={"symbols": symbols, **body})
|
||||||
|
|
||||||
|
def rolling_performance(self, symbol: str, **body: Any) -> dict:
|
||||||
|
"""Rolling (``window``) or exponentially weighted (``lam``) performance series."""
|
||||||
|
return self.post("/performance/rolling", json={"symbol": symbol, **body})
|
||||||
|
|
||||||
|
def portfolio_performance(self, holdings: dict[str, float], **body: Any) -> dict:
|
||||||
|
"""Performance of a share-based portfolio, rebalanced or buy-and-hold."""
|
||||||
|
return self.post("/performance/portfolio", json={"holdings": holdings, **body})
|
||||||
|
|
||||||
|
# -- models ---------------------------------------------------------------------------------
|
||||||
|
def expected_returns(self, **body: Any) -> dict:
|
||||||
|
"""Black-Litterman expected returns: equilibrium ``pi``, the posterior, and every
|
||||||
|
intermediate quantity (``delta``, ``tau``, the calibrated ``omega``). ``views`` are the same
|
||||||
|
objects ``/optimize`` takes, so a view can be inspected before it is optimised on."""
|
||||||
|
return self.post("/model/expected-returns", json=body)
|
||||||
|
|
||||||
|
def model_covariance(self, symbols: list[str], **body: Any) -> dict:
|
||||||
|
"""The covariance the scenario generator and the optimiser use, with its conditioning
|
||||||
|
(eigenvalues, applied shrinkage, effective observations)."""
|
||||||
|
return self.post("/model/covariance", json={"symbols": symbols, **body})
|
||||||
|
|
||||||
|
def scenarios(self, symbols: list[str], **body: Any) -> dict:
|
||||||
|
"""A scenario set and its statistics: per-stock moments, the P&L distribution of a
|
||||||
|
portfolio (``holdings`` or ``weights`` + ``value``), histograms, VaR/CVaR, and the
|
||||||
|
fingerprint (``hash``, ``shock_hash``) that identifies the draws."""
|
||||||
|
return self.post("/model/scenarios", json={"symbols": symbols, **body})
|
||||||
|
|
||||||
|
# -- optimisation ---------------------------------------------------------------------------
|
||||||
|
def optimize(self, **body: Any) -> dict:
|
||||||
|
"""Mean-CVaR MILP in integer shares. ``mode="new"`` invests ``cash``; ``mode="rebalance"``
|
||||||
|
starts from ``holdings`` and pays transaction ``costs``, and the result carries a ``before``
|
||||||
|
block so the two states can be compared.
|
||||||
|
|
||||||
|
The result names the solver, the status, the MIP gap, the reproducibility level, the
|
||||||
|
scenario key and hash, and the provenance of every model input. Can run for minutes: pass
|
||||||
|
``time_limit`` (seconds, the budget of the whole request) and a generous ``timeout=``.
|
||||||
|
"""
|
||||||
|
return self.post("/optimize", json=body)
|
||||||
|
|
||||||
|
# -- jobs and snapshots ---------------------------------------------------------------------
|
||||||
|
def start_setup(self, offline: bool = False, max_per_region: int | None = None, **kw: Any) -> dict:
|
||||||
|
"""Build the universe and download all history (~15 min). Returns a job id at once."""
|
||||||
|
return self.post("/jobs/setup", params={"offline": offline, "max_per_region": max_per_region}, **kw)
|
||||||
|
|
||||||
|
def start_update(self, full: bool = False, **kw: Any) -> dict:
|
||||||
|
"""Fetch the newest prices and FX rates. Returns a job id at once."""
|
||||||
|
return self.post("/jobs/update", params={"full": full}, **kw)
|
||||||
|
|
||||||
|
def jobs(self, **kw: Any) -> list[dict]:
|
||||||
|
return self.get("/jobs", **kw)
|
||||||
|
|
||||||
|
def job(self, job_id: str, **kw: Any) -> dict:
|
||||||
|
"""One job: ``queued`` | ``running`` | ``done`` | ``failed``, with its report or error."""
|
||||||
|
return self.get(f"/jobs/{job_id}", **kw)
|
||||||
|
|
||||||
|
def wait_for_job(self, job_id: str, poll: float = 5.0, deadline: float | None = None,
|
||||||
|
on_poll: Any = None) -> dict:
|
||||||
|
"""Poll a job until it is ``done`` or ``failed``; returns the final job record.
|
||||||
|
|
||||||
|
``deadline``: seconds to wait before giving up (``None``: forever). ``on_poll``: a callable
|
||||||
|
invoked with each intermediate record, for progress output.
|
||||||
|
"""
|
||||||
|
started = time.monotonic()
|
||||||
|
while True:
|
||||||
|
record = self.job(job_id)
|
||||||
|
if record.get("status") in JOB_STATES_DONE:
|
||||||
|
return record
|
||||||
|
if deadline is not None and time.monotonic() - started > deadline:
|
||||||
|
raise QuantcoreError(f"job {job_id} still {record.get('status')} after {deadline:.0f} s",
|
||||||
|
method="GET", path=f"/jobs/{job_id}")
|
||||||
|
if on_poll is not None:
|
||||||
|
on_poll(record)
|
||||||
|
time.sleep(poll)
|
||||||
|
|
||||||
|
def snapshots(self, **kw: Any) -> list[str]:
|
||||||
|
return self.get("/snapshots", **kw)
|
||||||
|
|
||||||
|
def create_snapshot(self, name: str | None = None, **kw: Any) -> dict:
|
||||||
|
"""Freeze universe, prices and FX under a name, for reproducible demos and grading."""
|
||||||
|
return self.post("/snapshots", json={"name": name}, **kw)
|
||||||
|
|
||||||
|
|
||||||
|
def _detail(res: Any) -> str:
|
||||||
|
"""The API's error message: FastAPI puts it in ``detail``, validation errors in a list."""
|
||||||
|
try:
|
||||||
|
body = res.json()
|
||||||
|
except Exception: # noqa: BLE001 (not JSON: a proxy error page, a truncated answer)
|
||||||
|
return (getattr(res, "text", "") or "")[:500]
|
||||||
|
detail = body.get("detail", body) if isinstance(body, dict) else body
|
||||||
|
if isinstance(detail, list): # 422: [{"loc": [...], "msg": ..., "type": ...}, ...]
|
||||||
|
return "; ".join(f"{'.'.join(str(p) for p in d.get('loc', []))}: {d.get('msg')}"
|
||||||
|
for d in detail if isinstance(d, dict))[:500]
|
||||||
|
return str(detail)[:500]
|
||||||
125
quantcore/config.py
Normal file
125
quantcore/config.py
Normal file
@ -0,0 +1,125 @@
|
|||||||
|
"""Central configuration (environment variables with the ``QUANTCORE_`` prefix, or a ``.env`` file).
|
||||||
|
|
||||||
|
Every service receives a :class:`Settings` instance -- nothing reads the environment on its own,
|
||||||
|
which keeps the services testable (tests pass a ``Settings(data_dir=tmp_path)``).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from functools import lru_cache
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from pydantic import Field
|
||||||
|
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||||
|
|
||||||
|
|
||||||
|
class Settings(BaseSettings):
|
||||||
|
"""Runtime settings of the quant core."""
|
||||||
|
|
||||||
|
model_config = SettingsConfigDict(env_prefix="QUANTCORE_", env_file=".env", extra="ignore")
|
||||||
|
|
||||||
|
# --- storage ---------------------------------------------------------------------------
|
||||||
|
data_dir: Path = Field(default=Path("data"), description="Root of all persisted data.")
|
||||||
|
|
||||||
|
# --- universe --------------------------------------------------------------------------
|
||||||
|
n_us: int = Field(default=500, description="Largest N US stocks (S&P 500 candidates).")
|
||||||
|
n_europe: int = Field(default=500, description="Largest N European stocks (STOXX 600 candidates).")
|
||||||
|
history_start: str = Field(default="2004-12-01", description="First date requested from Yahoo.")
|
||||||
|
required_history_from: str = Field(
|
||||||
|
default="2005-01-31",
|
||||||
|
description="A stock is kept only if it has a price on or before this date ('data since 2005').",
|
||||||
|
)
|
||||||
|
min_price_coverage: float = Field(
|
||||||
|
default=0.90, description="Minimum share of union-calendar days with a price since the first date."
|
||||||
|
)
|
||||||
|
max_missing_run: int = Field(
|
||||||
|
default=20, description="Longest tolerated gap (business days) in a price series since the required date."
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- market conventions ----------------------------------------------------------------
|
||||||
|
base_currency: str = Field(default="CHF", description="Portfolio / reporting currency.")
|
||||||
|
home_market: str = Field(default="Switzerland", description="Home market of the investor.")
|
||||||
|
risk_free_rate: float = Field(default=0.0, description="Annual risk-free rate in the base currency.")
|
||||||
|
trading_days_per_year: int = 252
|
||||||
|
weeks_per_year: int = 52
|
||||||
|
|
||||||
|
# --- Yahoo access ----------------------------------------------------------------------
|
||||||
|
yahoo_batch_size: int = Field(default=100, description="Tickers per bulk download request.")
|
||||||
|
yahoo_timeout: float = Field(default=30.0, description="Seconds per HTTP request.")
|
||||||
|
yahoo_max_retries: int = 4
|
||||||
|
yahoo_backoff: float = Field(default=2.0, description="Base of the exponential back-off (seconds).")
|
||||||
|
yahoo_info_pause: float = Field(default=0.15, description="Pause between ticker-info requests (s).")
|
||||||
|
|
||||||
|
# --- cuOpt -----------------------------------------------------------------------------
|
||||||
|
cuopt_url: str = Field(
|
||||||
|
default="http://127.0.0.1:5000",
|
||||||
|
description="cuOpt REST endpoint. The default is a cuOpt container on the same machine (the "
|
||||||
|
"standalone DGX Spark setup, deploy/apptainer/); point it elsewhere for a remote one.")
|
||||||
|
cuopt_time_limit: float = Field(default=120.0, description="MILP time limit in seconds.")
|
||||||
|
cuopt_poll_interval: float = 1.0
|
||||||
|
cuopt_timeout: float = Field(default=900.0, description="Max seconds to wait for a solution.")
|
||||||
|
cuopt_request_timeout: float = Field(default=60.0, description="Seconds per HTTP call to cuOpt.")
|
||||||
|
cuopt_upload_seconds_per_mb: float = Field(
|
||||||
|
default=1.0, description="Added to the submit timeout per MB of request body. cuOpt accepts about "
|
||||||
|
"10 MB/s, so this is a 10x margin; a client-side timeout during the upload "
|
||||||
|
"would otherwise surface as an aborted job.")
|
||||||
|
solver_fallback: bool = Field(
|
||||||
|
default=True, description="Fall back to the local HiGHS MILP solver (scipy) when cuOpt is unreachable."
|
||||||
|
)
|
||||||
|
cuopt_threads: int | None = Field(default=None, description="CPU threads of the cuOpt MIP solver (None: server default).")
|
||||||
|
cuopt_msgpack: bool = Field(default=True, description="Send cuOpt requests as msgpack when the package is available.")
|
||||||
|
|
||||||
|
# --- MILP tolerances (reproducibility) ---------------------------------------------------
|
||||||
|
mip_rel_gap: float = Field(default=1e-4, description="Relative MIP gap for HiGHS and cuOpt.")
|
||||||
|
mip_abs_gap: float = Field(default=0.01, description="Absolute MIP gap in money units (cuOpt).")
|
||||||
|
|
||||||
|
# --- misc ------------------------------------------------------------------------------
|
||||||
|
log_level: str = "INFO"
|
||||||
|
api_host: str = Field(
|
||||||
|
default="127.0.0.1",
|
||||||
|
description="Interface `quantcore serve` binds to: localhost only by default; the Docker images "
|
||||||
|
"pass 0.0.0.0 to publish the port.")
|
||||||
|
api_port: int = 5001
|
||||||
|
enable_ui: bool = Field(
|
||||||
|
default=True,
|
||||||
|
description="Serve the static test bench at /ui/ (and redirect / to it). Set false for a "
|
||||||
|
"deployment that only exposes the JSON API and /docs -- the same image then "
|
||||||
|
"serves a front end built by someone else.",
|
||||||
|
)
|
||||||
|
|
||||||
|
# derived paths -------------------------------------------------------------------------
|
||||||
|
@property
|
||||||
|
def universe_dir(self) -> Path:
|
||||||
|
return self.data_dir / "universe"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def prices_dir(self) -> Path:
|
||||||
|
return self.data_dir / "prices"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def fx_dir(self) -> Path:
|
||||||
|
return self.data_dir / "fx"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def snapshots_dir(self) -> Path:
|
||||||
|
return self.data_dir / "snapshots"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def jobs_dir(self) -> Path:
|
||||||
|
return self.data_dir / "jobs"
|
||||||
|
|
||||||
|
def ensure_dirs(self) -> None:
|
||||||
|
for d in (self.universe_dir, self.prices_dir, self.fx_dir, self.snapshots_dir, self.jobs_dir):
|
||||||
|
d.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=1)
|
||||||
|
def get_settings() -> Settings:
|
||||||
|
"""Process-wide settings (cached). Tests construct :class:`Settings` directly instead."""
|
||||||
|
settings = Settings()
|
||||||
|
logging.basicConfig(
|
||||||
|
level=getattr(logging, settings.log_level.upper(), logging.INFO),
|
||||||
|
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
||||||
|
)
|
||||||
|
return settings
|
||||||
18
quantcore/measurement/__init__.py
Normal file
18
quantcore/measurement/__init__.py
Normal file
@ -0,0 +1,18 @@
|
|||||||
|
"""Risk and performance measurement on log-returns (moving windows and exponential weighting)."""
|
||||||
|
|
||||||
|
from quantcore.measurement.performance import PerformanceService, performance_table
|
||||||
|
from quantcore.measurement.returns import (
|
||||||
|
FREQUENCIES,
|
||||||
|
HORIZONS,
|
||||||
|
align_prices,
|
||||||
|
horizon_periods,
|
||||||
|
log_returns,
|
||||||
|
periods_per_year,
|
||||||
|
resample_prices,
|
||||||
|
)
|
||||||
|
from quantcore.measurement.risk import RiskService, asset_risk_table
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"FREQUENCIES", "HORIZONS", "align_prices", "horizon_periods", "log_returns", "periods_per_year",
|
||||||
|
"resample_prices", "RiskService", "asset_risk_table", "PerformanceService", "performance_table",
|
||||||
|
]
|
||||||
228
quantcore/measurement/estimators.py
Normal file
228
quantcore/measurement/estimators.py
Normal file
@ -0,0 +1,228 @@
|
|||||||
|
"""Moving-window and exponentially weighted estimators of mean, volatility, covariance, correlation.
|
||||||
|
|
||||||
|
Two estimator families, both centred (the corresponding mean is subtracted):
|
||||||
|
|
||||||
|
* **rolling** -- equally weighted window of ``window`` observations (sample covariance, ddof=1);
|
||||||
|
* **ewm** -- exponentially weighted with decay ``lam`` (RiskMetrics: weight of the
|
||||||
|
observation ``k`` periods back is proportional to ``lam**k``; ``lam=0.94`` daily,
|
||||||
|
``lam=0.99`` for slow adaptation; effective window ~ 1/(1-lam)).
|
||||||
|
|
||||||
|
Per-asset *time series* use pandas (``rolling`` / ``ewm``); full matrices are computed for a
|
||||||
|
single date (``*_cov``) or streamed date by date (``covariance_sequence``) because a
|
||||||
|
``T x N x N`` array does not fit in memory for 1000 assets.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Iterator
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# per-asset time series
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def rolling_mean(r: pd.DataFrame | pd.Series, window: int, min_periods: int | None = None):
|
||||||
|
return r.rolling(window, min_periods=min_periods or max(2, window // 2)).mean()
|
||||||
|
|
||||||
|
|
||||||
|
def rolling_vol(r, window: int, periods_per_year: int | None = None, min_periods: int | None = None):
|
||||||
|
"""Rolling standard deviation, optionally annualised (x sqrt(periods per year))."""
|
||||||
|
vol = r.rolling(window, min_periods=min_periods or max(2, window // 2)).std(ddof=1)
|
||||||
|
return vol * np.sqrt(periods_per_year) if periods_per_year else vol
|
||||||
|
|
||||||
|
|
||||||
|
def ewm_mean(r, lam: float, min_periods: int = 10):
|
||||||
|
return r.ewm(alpha=1.0 - lam, adjust=True, min_periods=min_periods).mean()
|
||||||
|
|
||||||
|
|
||||||
|
def ewm_vol(r, lam: float, periods_per_year: int | None = None, min_periods: int = 10):
|
||||||
|
"""Exponentially weighted standard deviation (centred, weights normalised to one)."""
|
||||||
|
vol = np.sqrt(r.ewm(alpha=1.0 - lam, adjust=True, min_periods=min_periods).var(bias=True))
|
||||||
|
return vol * np.sqrt(periods_per_year) if periods_per_year else vol
|
||||||
|
|
||||||
|
|
||||||
|
def rolling_quantile(r, window: int, q: float, min_periods: int | None = None):
|
||||||
|
return r.rolling(window, min_periods=min_periods or max(2, window // 2)).quantile(q)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# matrices at one date
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def _window(r: pd.DataFrame, window: int | None, at=None) -> pd.DataFrame:
|
||||||
|
if at is not None:
|
||||||
|
r = r.loc[r.index <= pd.Timestamp(at)]
|
||||||
|
if window is not None:
|
||||||
|
r = r.iloc[-window:]
|
||||||
|
return r
|
||||||
|
|
||||||
|
|
||||||
|
def rolling_cov(r: pd.DataFrame, window: int, at=None, min_obs: int | None = None) -> pd.DataFrame:
|
||||||
|
"""Sample covariance of the last ``window`` rows (rows with any NaN are dropped)."""
|
||||||
|
sub = _window(r, window, at).dropna()
|
||||||
|
min_obs = min_obs or max(3, window // 2)
|
||||||
|
if len(sub) < min_obs:
|
||||||
|
raise ValueError(f"only {len(sub)} complete observations in the window (need {min_obs})")
|
||||||
|
# atleast_2d: np.cov of a single column returns a 0-d array, which DataFrame rejects -- a
|
||||||
|
# one-asset portfolio is a legitimate request (and the simplest one a caller makes)
|
||||||
|
cov = np.atleast_2d(np.cov(sub.values, rowvar=False, ddof=1))
|
||||||
|
return pd.DataFrame(cov, index=r.columns, columns=r.columns)
|
||||||
|
|
||||||
|
|
||||||
|
def ewm_weights(n: int, lam: float) -> np.ndarray:
|
||||||
|
"""Normalised weights for ``n`` observations, oldest first: ``lam**(n-1-t) / sum``."""
|
||||||
|
w = lam ** np.arange(n - 1, -1, -1, dtype=float)
|
||||||
|
return w / w.sum()
|
||||||
|
|
||||||
|
|
||||||
|
def ewm_cov(r: pd.DataFrame, lam: float, at=None, max_obs: int | None = None, min_obs: int = 20) -> pd.DataFrame:
|
||||||
|
"""Exponentially weighted covariance at ``at`` (default: last date), centred.
|
||||||
|
|
||||||
|
``max_obs`` truncates the history (weights below ``lam**max_obs`` are negligible; the
|
||||||
|
default keeps ``log(1e-4)/log(lam)`` observations, ~1500 for lam=0.994, ~150 for 0.94).
|
||||||
|
"""
|
||||||
|
max_obs = max_obs or int(np.ceil(np.log(1e-4) / np.log(lam)))
|
||||||
|
sub = _window(r, max_obs, at).dropna()
|
||||||
|
if len(sub) < min_obs:
|
||||||
|
raise ValueError(f"only {len(sub)} complete observations (need {min_obs})")
|
||||||
|
x = sub.values
|
||||||
|
w = ewm_weights(len(x), lam)
|
||||||
|
m = w @ x
|
||||||
|
xc = x - m
|
||||||
|
cov = (xc * w[:, None]).T @ xc
|
||||||
|
return pd.DataFrame(cov, index=r.columns, columns=r.columns)
|
||||||
|
|
||||||
|
|
||||||
|
def cov_to_corr(cov: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
sd = np.sqrt(np.diag(cov.values))
|
||||||
|
with np.errstate(divide="ignore", invalid="ignore"):
|
||||||
|
corr = cov.values / np.outer(sd, sd)
|
||||||
|
corr = np.clip(np.nan_to_num(corr), -1.0, 1.0)
|
||||||
|
np.fill_diagonal(corr, 1.0)
|
||||||
|
return pd.DataFrame(corr, index=cov.index, columns=cov.columns)
|
||||||
|
|
||||||
|
|
||||||
|
def rolling_corr(r: pd.DataFrame, window: int, at=None) -> pd.DataFrame:
|
||||||
|
return cov_to_corr(rolling_cov(r, window, at))
|
||||||
|
|
||||||
|
|
||||||
|
def ewm_corr(r: pd.DataFrame, lam: float, at=None) -> pd.DataFrame:
|
||||||
|
return cov_to_corr(ewm_cov(r, lam, at))
|
||||||
|
|
||||||
|
|
||||||
|
def shrinkage_intensity(x: np.ndarray) -> float:
|
||||||
|
"""Data-driven intensity of the shrinkage towards the diagonal (Schaefer & Strimmer 2005).
|
||||||
|
|
||||||
|
For centred observations ``x`` (n x N) the optimal linear shrinkage of the sample covariance
|
||||||
|
towards its diagonal is ``lambda* = sum_{i!=j} Var(s_ij) / sum_{i!=j} s_ij^2`` (clipped to
|
||||||
|
[0, 1]) -- the Ledoit-Wolf estimator with the "diagonal, unequal variances" target. It grows
|
||||||
|
with N / n, i.e. exactly when the sample covariance becomes ill-conditioned.
|
||||||
|
"""
|
||||||
|
x = np.asarray(x, dtype=float)
|
||||||
|
n = len(x)
|
||||||
|
if n < 3:
|
||||||
|
return 1.0
|
||||||
|
xc = x - x.mean(axis=0)
|
||||||
|
w_bar = xc.T @ xc / n # mean of the products w_kij
|
||||||
|
sq = (xc**2).T @ (xc**2) # sum_k w_kij^2
|
||||||
|
var_s = n / (n - 1) ** 3 * (sq - n * w_bar**2)
|
||||||
|
s = n / (n - 1) * w_bar
|
||||||
|
off = ~np.eye(len(s), dtype=bool)
|
||||||
|
denom = float((s[off] ** 2).sum())
|
||||||
|
if denom <= 0:
|
||||||
|
return 1.0
|
||||||
|
return float(np.clip(var_s[off].sum() / denom, 0.0, 1.0))
|
||||||
|
|
||||||
|
|
||||||
|
def effective_observations(method: str, window: int, lam: float) -> int:
|
||||||
|
return int(window) if method == "rolling" else int(round(3.0 / (1.0 - lam)))
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_shrinkage(shrinkage, r: pd.DataFrame, method: str, window: int, lam: float, at=None) -> float:
|
||||||
|
"""``'auto'`` -> Schaefer-Strimmer intensity on the (effective) estimation window; numbers pass through."""
|
||||||
|
if shrinkage in (None, 0, 0.0, "none"):
|
||||||
|
return 0.0
|
||||||
|
if isinstance(shrinkage, str):
|
||||||
|
if shrinkage != "auto":
|
||||||
|
raise ValueError("shrinkage must be a number in [0, 1] or 'auto'")
|
||||||
|
sub = _window(r, effective_observations(method, window, lam), at).dropna()
|
||||||
|
return shrinkage_intensity(sub.values)
|
||||||
|
return float(shrinkage)
|
||||||
|
|
||||||
|
|
||||||
|
def covariance(r: pd.DataFrame, method: str = "rolling", window: int = 200, lam: float = 0.94, at=None,
|
||||||
|
shrinkage=0.0) -> pd.DataFrame:
|
||||||
|
"""Dispatch on ``method`` (``'rolling'`` | ``'ewm'``) with optional shrinkage to the diagonal.
|
||||||
|
|
||||||
|
Linear shrinkage ``(1-s) * Sigma + s * diag(Sigma)`` keeps the variances and pulls the
|
||||||
|
correlations towards zero -- needed when N is not small against the effective window.
|
||||||
|
``shrinkage='auto'`` estimates the intensity from the data (:func:`shrinkage_intensity`).
|
||||||
|
"""
|
||||||
|
if method == "rolling":
|
||||||
|
cov = rolling_cov(r, window, at)
|
||||||
|
elif method == "ewm":
|
||||||
|
cov = ewm_cov(r, lam, at)
|
||||||
|
else:
|
||||||
|
raise ValueError("method must be 'rolling' or 'ewm'")
|
||||||
|
return shrink(cov, resolve_shrinkage(shrinkage, r, method, window, lam, at))
|
||||||
|
|
||||||
|
|
||||||
|
def shrink(cov: pd.DataFrame, intensity: float) -> pd.DataFrame:
|
||||||
|
if intensity <= 0:
|
||||||
|
return cov
|
||||||
|
if not 0 < intensity <= 1:
|
||||||
|
raise ValueError("shrinkage intensity must be in [0, 1]")
|
||||||
|
diag = np.diag(np.diag(cov.values))
|
||||||
|
return pd.DataFrame((1 - intensity) * cov.values + intensity * diag, index=cov.index, columns=cov.columns)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# streamed sequences of matrices (for the innovation filter)
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def covariance_sequence(
|
||||||
|
r: pd.DataFrame, method: str, window: int = 200, lam: float = 0.94, warmup: int | None = None,
|
||||||
|
shrinkage=0.0,
|
||||||
|
) -> Iterator[tuple[pd.Timestamp, np.ndarray, np.ndarray]]:
|
||||||
|
"""Yield ``(date_t, Sigma_{t-1}, r_t)`` for every ``t`` after the warm-up.
|
||||||
|
|
||||||
|
``Sigma_{t-1}`` uses data **up to and including t-1** -- the information set for
|
||||||
|
standardising the return of day ``t``. The rolling estimator recomputes the window at
|
||||||
|
every step (O(w N^2) each); the EWM estimator uses the exact recursion of the centred
|
||||||
|
RiskMetrics filter, so the cost per step is O(N^2).
|
||||||
|
"""
|
||||||
|
x = r.dropna().values.astype(float)
|
||||||
|
dates = r.dropna().index
|
||||||
|
n_obs, n = x.shape
|
||||||
|
shrinkage = resolve_shrinkage(shrinkage, r, method, window, lam) # one intensity for the whole sequence
|
||||||
|
if method == "rolling":
|
||||||
|
warmup = warmup or window
|
||||||
|
if n_obs <= warmup:
|
||||||
|
raise ValueError("not enough observations for the warm-up window")
|
||||||
|
for t in range(warmup, n_obs):
|
||||||
|
block = x[t - window : t]
|
||||||
|
cov = np.atleast_2d(np.cov(block, rowvar=False, ddof=1))
|
||||||
|
yield dates[t], shrink_array(cov, shrinkage), x[t]
|
||||||
|
elif method == "ewm":
|
||||||
|
warmup = warmup or min(n_obs - 1, max(50, int(2 / (1 - lam))))
|
||||||
|
if n_obs <= warmup:
|
||||||
|
raise ValueError("not enough observations for the warm-up window")
|
||||||
|
# initialise with the normalised-weight estimate over the warm-up block, then recurse
|
||||||
|
w = ewm_weights(warmup, lam)
|
||||||
|
m = w @ x[:warmup]
|
||||||
|
xc = x[:warmup] - m
|
||||||
|
cov = (xc * w[:, None]).T @ xc
|
||||||
|
alpha = 1.0 - lam
|
||||||
|
for t in range(warmup, n_obs):
|
||||||
|
yield dates[t], shrink_array(cov, shrinkage), x[t]
|
||||||
|
d = x[t] - m
|
||||||
|
m = m + alpha * d
|
||||||
|
cov = lam * cov + alpha * np.outer(d, x[t] - m) # Welford-style centred update
|
||||||
|
else:
|
||||||
|
raise ValueError("method must be 'rolling' or 'ewm'")
|
||||||
|
|
||||||
|
|
||||||
|
def shrink_array(cov: np.ndarray, intensity: float) -> np.ndarray:
|
||||||
|
if intensity <= 0:
|
||||||
|
return cov
|
||||||
|
return (1 - intensity) * cov + intensity * np.diag(np.diag(cov))
|
||||||
145
quantcore/measurement/performance.py
Normal file
145
quantcore/measurement/performance.py
Normal file
@ -0,0 +1,145 @@
|
|||||||
|
"""Performance measurement on log-returns: return, volatility, Sharpe, drawdown and friends.
|
||||||
|
|
||||||
|
All statistics are computed from one-period log-returns at a given frequency and annualised
|
||||||
|
with the number of periods per year (252 / 52 / 12). ``sharpe`` uses the annual risk-free rate
|
||||||
|
of the base currency. Rolling and exponentially weighted variants give time series.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
from scipy import stats
|
||||||
|
|
||||||
|
from quantcore.measurement import estimators as est
|
||||||
|
from quantcore.measurement.returns import periods_per_year, portfolio_log_returns
|
||||||
|
|
||||||
|
|
||||||
|
def cumulative_return(r: pd.DataFrame | pd.Series):
|
||||||
|
"""Total simple return over the sample: ``exp(sum r) - 1``."""
|
||||||
|
return np.expm1(r.sum())
|
||||||
|
|
||||||
|
|
||||||
|
def annualised_return(r, ppy: int):
|
||||||
|
"""Annualised log-return (arithmetic mean of log-returns x periods per year)."""
|
||||||
|
return r.mean() * ppy
|
||||||
|
|
||||||
|
|
||||||
|
def annualised_volatility(r, ppy: int):
|
||||||
|
return r.std(ddof=1) * np.sqrt(ppy)
|
||||||
|
|
||||||
|
|
||||||
|
def sharpe_ratio(r, ppy: int, risk_free: float = 0.0):
|
||||||
|
vol = annualised_volatility(r, ppy)
|
||||||
|
return (annualised_return(r, ppy) - risk_free) / vol
|
||||||
|
|
||||||
|
|
||||||
|
def sortino_ratio(r, ppy: int, risk_free: float = 0.0):
|
||||||
|
downside = r.where(r < 0, 0.0)
|
||||||
|
dd = np.sqrt((downside**2).mean() * ppy)
|
||||||
|
return (annualised_return(r, ppy) - risk_free) / dd
|
||||||
|
|
||||||
|
|
||||||
|
def drawdown_series(r: pd.Series | pd.DataFrame):
|
||||||
|
"""Drawdown (fraction below the running peak) of the cumulative value path."""
|
||||||
|
wealth = np.exp(r.fillna(0.0).cumsum())
|
||||||
|
return wealth / wealth.cummax() - 1.0
|
||||||
|
|
||||||
|
|
||||||
|
def max_drawdown(r):
|
||||||
|
return drawdown_series(r).min()
|
||||||
|
|
||||||
|
|
||||||
|
def calmar_ratio(r, ppy: int):
|
||||||
|
mdd = max_drawdown(r)
|
||||||
|
ann = np.expm1(annualised_return(r, ppy)) # geometric annual return
|
||||||
|
return ann / -mdd if np.ndim(mdd) == 0 else ann / -mdd
|
||||||
|
|
||||||
|
|
||||||
|
def performance_table(r: pd.DataFrame, freq: str = "daily", risk_free: float = 0.0) -> pd.DataFrame:
|
||||||
|
"""One row per column of ``r`` with the standard performance statistics."""
|
||||||
|
ppy = periods_per_year(freq)
|
||||||
|
out = pd.DataFrame(index=r.columns)
|
||||||
|
out["start"] = r.apply(lambda c: c.first_valid_index())
|
||||||
|
out["end"] = r.index[-1]
|
||||||
|
out["n_obs"] = r.notna().sum()
|
||||||
|
out["cumulative_return"] = cumulative_return(r)
|
||||||
|
out["annualised_return_log"] = annualised_return(r, ppy)
|
||||||
|
out["annualised_return"] = np.expm1(out["annualised_return_log"])
|
||||||
|
out["annualised_volatility"] = annualised_volatility(r, ppy)
|
||||||
|
out["sharpe"] = sharpe_ratio(r, ppy, risk_free)
|
||||||
|
out["sortino"] = sortino_ratio(r, ppy, risk_free)
|
||||||
|
out["max_drawdown"] = max_drawdown(r)
|
||||||
|
out["calmar"] = np.expm1(out["annualised_return_log"]) / -out["max_drawdown"]
|
||||||
|
out["skewness"] = r.apply(lambda c: stats.skew(c.dropna()))
|
||||||
|
out["excess_kurtosis"] = r.apply(lambda c: stats.kurtosis(c.dropna()))
|
||||||
|
out["hit_ratio"] = (r > 0).sum() / r.notna().sum()
|
||||||
|
out["best_period"] = r.max()
|
||||||
|
out["worst_period"] = r.min()
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def rolling_performance(r: pd.Series, window: int, freq: str = "daily", risk_free: float = 0.0) -> pd.DataFrame:
|
||||||
|
"""Rolling annualised return, volatility, Sharpe and the running drawdown."""
|
||||||
|
ppy = periods_per_year(freq)
|
||||||
|
df = pd.DataFrame(index=r.index)
|
||||||
|
mean = est.rolling_mean(r, window)
|
||||||
|
vol = est.rolling_vol(r, window)
|
||||||
|
df["return_ann"] = mean * ppy
|
||||||
|
df["volatility_ann"] = vol * np.sqrt(ppy)
|
||||||
|
df["sharpe"] = (df["return_ann"] - risk_free) / df["volatility_ann"]
|
||||||
|
df["cumulative_window"] = np.expm1(r.rolling(window, min_periods=window // 2).sum())
|
||||||
|
df["drawdown"] = drawdown_series(r)
|
||||||
|
return df
|
||||||
|
|
||||||
|
|
||||||
|
def ewm_performance(r: pd.Series, lam: float, freq: str = "daily", risk_free: float = 0.0) -> pd.DataFrame:
|
||||||
|
ppy = periods_per_year(freq)
|
||||||
|
df = pd.DataFrame(index=r.index)
|
||||||
|
df["return_ann"] = est.ewm_mean(r, lam) * ppy
|
||||||
|
df["volatility_ann"] = est.ewm_vol(r, lam) * np.sqrt(ppy)
|
||||||
|
df["sharpe"] = (df["return_ann"] - risk_free) / df["volatility_ann"]
|
||||||
|
df["drawdown"] = drawdown_series(r)
|
||||||
|
return df
|
||||||
|
|
||||||
|
|
||||||
|
class PerformanceService:
|
||||||
|
"""Deterministic performance measurement for assets and share-based portfolios."""
|
||||||
|
|
||||||
|
def __init__(self, market_data, risk_free: float = 0.0):
|
||||||
|
self.md = market_data
|
||||||
|
self.risk_free = risk_free
|
||||||
|
|
||||||
|
def asset_performance(self, symbols, freq="daily", currency="base", start=None, end=None) -> pd.DataFrame:
|
||||||
|
r = self.md.returns(symbols, start=start, end=end, currency=currency, freq=freq)
|
||||||
|
return performance_table(r, freq, self.risk_free)
|
||||||
|
|
||||||
|
def rolling(self, symbol, window=252, freq="daily", currency="base", start=None, end=None) -> pd.DataFrame:
|
||||||
|
r = self.md.returns([symbol], start=start, end=end, currency=currency, freq=freq)[symbol].dropna()
|
||||||
|
return rolling_performance(r, window, freq, self.risk_free)
|
||||||
|
|
||||||
|
def ewm(self, symbol, lam=0.99, freq="daily", currency="base", start=None, end=None) -> pd.DataFrame:
|
||||||
|
r = self.md.returns([symbol], start=start, end=end, currency=currency, freq=freq)[symbol].dropna()
|
||||||
|
return ewm_performance(r, lam, freq, self.risk_free)
|
||||||
|
|
||||||
|
def portfolio_returns(self, holdings: dict[str, float], freq="daily", currency="base", start=None, end=None,
|
||||||
|
rebalanced: bool = True) -> pd.Series:
|
||||||
|
"""Log-returns of a share-based portfolio.
|
||||||
|
|
||||||
|
``rebalanced=True`` keeps today's value weights fixed (a constant-mix benchmark);
|
||||||
|
``False`` is buy-and-hold with the given share numbers over the whole history.
|
||||||
|
"""
|
||||||
|
symbols = list(holdings)
|
||||||
|
px = self.md.prices(symbols, start=start, end=end, currency=currency, freq=freq)
|
||||||
|
shares = pd.Series(holdings, dtype=float).reindex(symbols)
|
||||||
|
if rebalanced:
|
||||||
|
values = shares * px.ffill().iloc[-1]
|
||||||
|
return portfolio_log_returns(np.log(px).diff().iloc[1:], (values / values.sum()).values)
|
||||||
|
value = (px.ffill() * shares).sum(axis=1)
|
||||||
|
return np.log(value).diff().iloc[1:].rename("portfolio")
|
||||||
|
|
||||||
|
def portfolio_performance(self, holdings, freq="daily", currency="base", start=None, end=None,
|
||||||
|
rebalanced=True) -> pd.Series:
|
||||||
|
rp = self.portfolio_returns(holdings, freq, currency, start, end, rebalanced)
|
||||||
|
table = performance_table(rp.to_frame("portfolio"), freq, self.risk_free)
|
||||||
|
return table.iloc[0]
|
||||||
116
quantcore/measurement/returns.py
Normal file
116
quantcore/measurement/returns.py
Normal file
@ -0,0 +1,116 @@
|
|||||||
|
"""Log-returns at daily / weekly / monthly frequency, calendar alignment and horizons.
|
||||||
|
|
||||||
|
Conventions used throughout the package:
|
||||||
|
|
||||||
|
* prices are adjusted closes, one column per symbol, indexed by calendar date (tz-naive);
|
||||||
|
* the **union calendar** of all listings is used; a holiday on one exchange is filled with
|
||||||
|
the previous close (``align_prices``), which yields a zero return on that day;
|
||||||
|
* ``log_returns`` are ``ln(P_t / P_{t-1})``; a CHF return of a foreign stock is the local
|
||||||
|
log-return plus the FX log-return (see :mod:`quantcore.data.fx`);
|
||||||
|
* weekly returns are Friday-to-Friday, monthly returns month-end to month-end;
|
||||||
|
* a *horizon* (``'day'``, ``'week'``, ``'month'`` or an integer) is expressed in periods of the
|
||||||
|
chosen frequency (``horizon_periods``), assuming i.i.d. returns that add up over time.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
FREQUENCIES = {"daily": 252, "weekly": 52, "monthly": 12}
|
||||||
|
HORIZONS_DAYS = {"day": 1, "week": 5, "month": 21, "quarter": 63, "year": 252}
|
||||||
|
HORIZONS = list(HORIZONS_DAYS)
|
||||||
|
|
||||||
|
|
||||||
|
def periods_per_year(freq: str) -> int:
|
||||||
|
try:
|
||||||
|
return FREQUENCIES[freq]
|
||||||
|
except KeyError as exc:
|
||||||
|
raise ValueError(f"unknown frequency {freq!r}; use one of {list(FREQUENCIES)}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def horizon_periods(horizon: str | int, freq: str = "daily") -> int:
|
||||||
|
"""Number of ``freq`` periods in ``horizon`` (``'week'`` at daily frequency -> 5)."""
|
||||||
|
if isinstance(horizon, (int, np.integer)):
|
||||||
|
if horizon < 1:
|
||||||
|
raise ValueError("horizon must be >= 1 period")
|
||||||
|
return int(horizon)
|
||||||
|
try:
|
||||||
|
days = HORIZONS_DAYS[horizon]
|
||||||
|
except KeyError as exc:
|
||||||
|
raise ValueError(f"unknown horizon {horizon!r}; use one of {HORIZONS} or an integer") from exc
|
||||||
|
periods = days * periods_per_year(freq) / FREQUENCIES["daily"]
|
||||||
|
if periods < 1:
|
||||||
|
raise ValueError(f"horizon {horizon!r} is shorter than one {freq} period")
|
||||||
|
return int(round(periods))
|
||||||
|
|
||||||
|
|
||||||
|
def align_prices(prices: pd.DataFrame, max_ffill: int | None = 5) -> pd.DataFrame:
|
||||||
|
"""Union calendar with limited forward-fill of exchange holidays; leading NaNs stay NaN."""
|
||||||
|
prices = prices.sort_index()
|
||||||
|
# keep business days only (Yahoo occasionally returns a weekend row for some exchanges)
|
||||||
|
prices = prices[prices.index.dayofweek < 5]
|
||||||
|
return prices.ffill(limit=max_ffill) if max_ffill else prices
|
||||||
|
|
||||||
|
|
||||||
|
def _month_end_alias() -> str:
|
||||||
|
"""``'ME'`` from pandas 2.2 on, ``'M'`` before it (pandas 3 dropped the old spelling)."""
|
||||||
|
for alias in ("ME", "M"):
|
||||||
|
try:
|
||||||
|
pd.tseries.frequencies.to_offset(alias)
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
return alias
|
||||||
|
raise RuntimeError("no month-end frequency alias accepted by this pandas")
|
||||||
|
|
||||||
|
|
||||||
|
MONTH_END = _month_end_alias()
|
||||||
|
|
||||||
|
|
||||||
|
def resample_prices(prices: pd.DataFrame, freq: str) -> pd.DataFrame:
|
||||||
|
"""Daily prices -> the last observation of each week (Fri) or month."""
|
||||||
|
if freq == "daily":
|
||||||
|
return prices
|
||||||
|
if freq == "weekly":
|
||||||
|
return prices.resample("W-FRI").last()
|
||||||
|
if freq == "monthly":
|
||||||
|
return prices.resample(MONTH_END).last()
|
||||||
|
raise ValueError(f"unknown frequency {freq!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def log_returns(prices: pd.DataFrame | pd.Series) -> pd.DataFrame | pd.Series:
|
||||||
|
"""``ln(P_t / P_{t-1})``; the first row (all NaN) is dropped."""
|
||||||
|
r = np.log(prices).diff()
|
||||||
|
return r.iloc[1:] if len(r) else r
|
||||||
|
|
||||||
|
|
||||||
|
def simple_returns(prices: pd.DataFrame | pd.Series) -> pd.DataFrame | pd.Series:
|
||||||
|
return prices.pct_change().iloc[1:]
|
||||||
|
|
||||||
|
|
||||||
|
def horizon_returns(r: pd.DataFrame | pd.Series, periods: int, overlapping: bool = True):
|
||||||
|
"""Aggregate one-period log-returns to ``periods``-period log-returns (sums).
|
||||||
|
|
||||||
|
Overlapping windows keep ``T - periods + 1`` observations (more data, autocorrelated);
|
||||||
|
non-overlapping blocks are independent under the i.i.d. assumption but fewer.
|
||||||
|
"""
|
||||||
|
if periods == 1:
|
||||||
|
return r
|
||||||
|
if overlapping:
|
||||||
|
return r.rolling(periods).sum().iloc[periods - 1 :]
|
||||||
|
blocks = np.arange(len(r))[::-1] // periods # block id counted from the end
|
||||||
|
grouped = r.groupby(blocks[::-1]).sum()
|
||||||
|
counts = r.groupby(blocks[::-1]).size()
|
||||||
|
return grouped[counts == periods]
|
||||||
|
|
||||||
|
|
||||||
|
def portfolio_log_returns(r: pd.DataFrame, weights: pd.Series | np.ndarray) -> pd.Series:
|
||||||
|
"""Log-return of a portfolio with weights fixed at the start of each period.
|
||||||
|
|
||||||
|
Exact aggregation: the simple portfolio return is the weighted sum of simple returns,
|
||||||
|
``ln(1 + sum_i w_i (exp(r_i) - 1))``. Weights are normalised to the invested fraction; a
|
||||||
|
cash remainder (``1 - sum w``) earns zero.
|
||||||
|
"""
|
||||||
|
w = np.asarray(weights, dtype=float)
|
||||||
|
simple = np.expm1(r.fillna(0.0).values) @ w
|
||||||
|
return pd.Series(np.log1p(simple), index=r.index, name="portfolio")
|
||||||
401
quantcore/measurement/risk.py
Normal file
401
quantcore/measurement/risk.py
Normal file
@ -0,0 +1,401 @@
|
|||||||
|
"""Value-at-Risk and Conditional Value-at-Risk (expected shortfall) on log-returns.
|
||||||
|
|
||||||
|
Conventions
|
||||||
|
-----------
|
||||||
|
* Losses are positive numbers: for a log-return ``r`` over the horizon, the loss in
|
||||||
|
return space is ``-r``; in money it is ``V0 * (1 - exp(r))``.
|
||||||
|
* ``VaR_beta`` is the ``beta``-quantile of the loss distribution, ``CVaR_beta`` the expected
|
||||||
|
loss beyond it (``beta`` = 0.95 or 0.99).
|
||||||
|
* Horizons: i.i.d. returns scale the mean with ``h`` and the volatility with ``sqrt(h)``
|
||||||
|
(parametric); the historical method aggregates ``h`` consecutive returns.
|
||||||
|
|
||||||
|
Methods
|
||||||
|
-------
|
||||||
|
* ``normal`` -- Gaussian (GBM) with the chosen volatility estimator;
|
||||||
|
* ``historical`` -- empirical quantiles of the (window of) realised returns;
|
||||||
|
* time series of both, driven by moving-window or exponentially weighted volatilities.
|
||||||
|
|
||||||
|
Money amounts ("an absolute level for a period") use the exact mapping of a log-return to a
|
||||||
|
P&L, so a 99 % CVaR of CHF 12 000 over a week reads: *in the worst 1 % of weeks the average
|
||||||
|
loss is CHF 12 000*.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
from scipy import stats
|
||||||
|
|
||||||
|
from quantcore.measurement import estimators as est
|
||||||
|
from quantcore.measurement.returns import horizon_periods, horizon_returns, periods_per_year, portfolio_log_returns
|
||||||
|
|
||||||
|
DEFAULT_CONFIDENCE = (0.95, 0.99)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# closed forms and empirical estimators (return space, positive losses)
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def normal_var(mu: float | np.ndarray, sigma: float | np.ndarray, beta: float):
|
||||||
|
"""Gaussian VaR of the loss ``-r`` with ``r ~ N(mu, sigma^2)``."""
|
||||||
|
return -mu + stats.norm.ppf(beta) * sigma
|
||||||
|
|
||||||
|
|
||||||
|
def normal_cvar(mu, sigma, beta: float):
|
||||||
|
"""Gaussian expected shortfall: ``-mu + sigma * phi(z_beta) / (1 - beta)``."""
|
||||||
|
z = stats.norm.ppf(beta)
|
||||||
|
return -mu + sigma * stats.norm.pdf(z) / (1.0 - beta)
|
||||||
|
|
||||||
|
|
||||||
|
def normal_value_var(mu, sigma, beta: float):
|
||||||
|
"""VaR as a *fraction of the position's value* when the log-return is ``N(mu, sigma^2)``.
|
||||||
|
|
||||||
|
A quantile transforms exactly under the monotone map ``r -> 1 - e^r``, so this is simply
|
||||||
|
``1 - exp(-VaR_log)`` -- no Ito term is involved.
|
||||||
|
"""
|
||||||
|
return 1.0 - np.exp(-normal_var(mu, sigma, beta))
|
||||||
|
|
||||||
|
|
||||||
|
def normal_value_cvar(mu, sigma, beta: float):
|
||||||
|
"""CVaR as a fraction of the position's value for a lognormal price.
|
||||||
|
|
||||||
|
A CVaR is a tail *mean*, and ``1 - e^{-x}`` is concave, so ``1 - exp(-CVaR_log)`` is not the
|
||||||
|
expected value loss -- it overstates it (0.7 % relative at sigma = 0.30). The exact figure uses
|
||||||
|
the truncated lognormal moment ``E[e^r 1{r <= q}] = e^{mu + sigma^2/2} Phi(z_{1-beta} - sigma)``::
|
||||||
|
|
||||||
|
CVaR_value = 1 - e^{mu + sigma^2/2} Phi(z_{1-beta} - sigma) / (1 - beta)
|
||||||
|
|
||||||
|
which is the same ``sigma^2/2`` correction that turns a log drift into an arithmetic one.
|
||||||
|
"""
|
||||||
|
mu = np.asarray(mu, dtype=float)
|
||||||
|
sigma = np.asarray(sigma, dtype=float)
|
||||||
|
z = stats.norm.ppf(1.0 - beta)
|
||||||
|
return 1.0 - np.exp(mu + 0.5 * sigma ** 2) * stats.norm.cdf(z - sigma) / (1.0 - beta)
|
||||||
|
|
||||||
|
|
||||||
|
def historical_var(r: pd.Series | pd.DataFrame | np.ndarray, beta: float):
|
||||||
|
"""Empirical VaR: minus the ``(1 - beta)``-quantile of the returns."""
|
||||||
|
if isinstance(r, (pd.Series, pd.DataFrame)):
|
||||||
|
return -r.quantile(1.0 - beta)
|
||||||
|
return -np.quantile(np.asarray(r), 1.0 - beta, axis=0)
|
||||||
|
|
||||||
|
|
||||||
|
def historical_cvar(r: pd.Series | pd.DataFrame | np.ndarray, beta: float):
|
||||||
|
"""Empirical CVaR: minus the mean of the returns at or below the ``(1 - beta)``-quantile."""
|
||||||
|
if isinstance(r, pd.DataFrame):
|
||||||
|
return pd.Series({c: historical_cvar(r[c].dropna(), beta) for c in r.columns})
|
||||||
|
x = np.asarray(r.dropna() if isinstance(r, pd.Series) else r, dtype=float)
|
||||||
|
if x.ndim == 2:
|
||||||
|
return np.array([historical_cvar(x[:, j], beta) for j in range(x.shape[1])])
|
||||||
|
if len(x) == 0:
|
||||||
|
return np.nan
|
||||||
|
q = np.quantile(x, 1.0 - beta)
|
||||||
|
tail = x[x <= q]
|
||||||
|
return -tail.mean() if len(tail) else -q
|
||||||
|
|
||||||
|
|
||||||
|
def scenario_cvar(losses: np.ndarray, beta: float, probabilities: np.ndarray | None = None) -> tuple[float, float]:
|
||||||
|
"""(VaR, CVaR) of a discrete loss sample, consistent with the Rockafellar-Uryasev LP.
|
||||||
|
|
||||||
|
With ``S`` equiprobable scenarios the CVaR is the mean of the ``(1-beta)S`` largest losses
|
||||||
|
(fractional tail weight handled exactly), which the LP formulation reproduces. With scenario
|
||||||
|
``probabilities`` (importance sampling) the tail is the set of largest losses whose probability
|
||||||
|
adds up to ``1-beta``, the last one taken fractionally.
|
||||||
|
"""
|
||||||
|
losses = np.asarray(losses, dtype=float)
|
||||||
|
if probabilities is None:
|
||||||
|
losses = np.sort(losses)[::-1]
|
||||||
|
s = len(losses)
|
||||||
|
k = (1.0 - beta) * s
|
||||||
|
n_full = int(np.floor(k + 1e-9)) # tolerance: (1 - 0.95) * 1000 is 50.00000000000004 in floating point
|
||||||
|
frac = k - n_full if k - n_full > 1e-9 else 0.0
|
||||||
|
var = losses[min(n_full, s - 1)] if frac > 0 or n_full == 0 else losses[n_full - 1]
|
||||||
|
tail = losses[:n_full].sum() + (frac * losses[n_full] if n_full < s else 0.0)
|
||||||
|
return float(var), float(tail / k) if k > 0 else float(losses[0])
|
||||||
|
p = np.asarray(probabilities, dtype=float)
|
||||||
|
p = p / p.sum()
|
||||||
|
order = np.argsort(losses)[::-1]
|
||||||
|
sorted_l, w = losses[order], p[order]
|
||||||
|
tail_mass = 1.0 - beta
|
||||||
|
cum = np.cumsum(w)
|
||||||
|
k = int(np.searchsorted(cum, tail_mass - 1e-12, side="left"))
|
||||||
|
k = min(k, len(sorted_l) - 1)
|
||||||
|
var = float(sorted_l[k])
|
||||||
|
full_mass = cum[k - 1] if k > 0 else 0.0
|
||||||
|
frac = max(tail_mass - full_mass, 0.0)
|
||||||
|
cvar = (sorted_l[:k] @ w[:k] + frac * sorted_l[k]) / tail_mass
|
||||||
|
return var, float(cvar)
|
||||||
|
|
||||||
|
|
||||||
|
def weighted_quantile(x, q, weights: np.ndarray | None = None) -> np.ndarray:
|
||||||
|
"""Quantiles of a sample that carries probability ``weights`` (importance sampling).
|
||||||
|
|
||||||
|
Column-wise for a 2-D ``x``. Without weights this is ``np.quantile``; with them the weighted
|
||||||
|
CDF is interpolated at ``c_i = (sum_{j<=i} w_j - w_i / 2) / sum w``, the usual convention that
|
||||||
|
reproduces the unweighted result for equal weights.
|
||||||
|
"""
|
||||||
|
x = np.asarray(x, dtype=float)
|
||||||
|
q = np.atleast_1d(np.asarray(q, dtype=float))
|
||||||
|
flat = x.ndim == 1
|
||||||
|
xx = x[:, None] if flat else x
|
||||||
|
if weights is None:
|
||||||
|
out = np.quantile(xx, q, axis=0)
|
||||||
|
else:
|
||||||
|
w = np.asarray(weights, dtype=float)
|
||||||
|
order = np.argsort(xx, axis=0)
|
||||||
|
sorted_x = np.take_along_axis(xx, order, axis=0)
|
||||||
|
sorted_w = w[order]
|
||||||
|
cum = np.cumsum(sorted_w, axis=0) - 0.5 * sorted_w
|
||||||
|
cum = cum / sorted_w.sum(axis=0)
|
||||||
|
out = np.stack([[np.interp(qq, cum[:, j], sorted_x[:, j]) for j in range(xx.shape[1])] for qq in q])
|
||||||
|
return out[:, 0] if flat else out
|
||||||
|
|
||||||
|
|
||||||
|
def loss_to_money(loss_return, value: float):
|
||||||
|
"""Exact money loss for a loss in log-return space: ``V0 * (1 - exp(-loss))``."""
|
||||||
|
return value * (1.0 - np.exp(-loss_return))
|
||||||
|
|
||||||
|
|
||||||
|
def value_loss(loss_return):
|
||||||
|
"""A loss in log-return space as a fraction of the position's value: ``1 - exp(-x)``.
|
||||||
|
|
||||||
|
A VaR of 0.30 in log terms is a loss of 25.9 % of the value, never 30 %; the difference is
|
||||||
|
negligible for a day and material for a quarter. It can never reach 100 %.
|
||||||
|
"""
|
||||||
|
return 1.0 - np.exp(-np.asarray(loss_return, dtype=float))
|
||||||
|
|
||||||
|
|
||||||
|
RETURN_SPACES = ("log", "value")
|
||||||
|
DRIFT_MODES = ("estimated", "zero")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# per-asset table at one date
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def asset_risk_table(
|
||||||
|
r: pd.DataFrame,
|
||||||
|
freq: str = "daily",
|
||||||
|
window: int = 200,
|
||||||
|
lam: float = 0.94,
|
||||||
|
confidence=DEFAULT_CONFIDENCE,
|
||||||
|
horizon: str | int = "day",
|
||||||
|
hist_window: int | None = None,
|
||||||
|
at=None,
|
||||||
|
space: str = "log",
|
||||||
|
drift: str = "estimated",
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Volatility, VaR and CVaR per asset for one horizon (rows: assets).
|
||||||
|
|
||||||
|
Columns: ``vol_rolling_ann``, ``vol_ewm_ann`` (annualised), then for every confidence
|
||||||
|
level ``var{cc}_normal_rolling``, ``cvar{cc}_normal_rolling``, ``var{cc}_normal_ewm``,
|
||||||
|
``cvar{cc}_normal_ewm``, ``var{cc}_hist``, ``cvar{cc}_hist`` (positive = loss).
|
||||||
|
|
||||||
|
``space='log'`` reports them in log-return units (the modelling space); ``'value'`` as a
|
||||||
|
fraction of the position's value, which cannot reach 100 %. The value figures are *computed*
|
||||||
|
in that space -- the normal ones from the lognormal closed forms, the historical ones from the
|
||||||
|
simple returns of the same window -- because only the VaR survives the map ``1 - e^{-x}``
|
||||||
|
unchanged; a CVaR is a tail mean and would be overstated by converting it afterwards.
|
||||||
|
|
||||||
|
``drift='estimated'`` carries the window's mean into the parametric VaR/CVaR (``-mu h``);
|
||||||
|
harmless for a day, dominant for a quarter, where the mean is estimated far less precisely
|
||||||
|
than the volatility -- a stock in a strong rally then shows a *smaller* VaR than a quiet one.
|
||||||
|
``'zero'`` is the usual market-risk convention: volatility only. The historical columns always
|
||||||
|
carry the drift of their window.
|
||||||
|
"""
|
||||||
|
if space not in RETURN_SPACES:
|
||||||
|
raise ValueError(f"space must be one of {RETURN_SPACES}")
|
||||||
|
if drift not in DRIFT_MODES:
|
||||||
|
raise ValueError(f"drift must be one of {DRIFT_MODES}")
|
||||||
|
ppy = periods_per_year(freq)
|
||||||
|
h = horizon_periods(horizon, freq)
|
||||||
|
if at is not None:
|
||||||
|
r = r.loc[r.index <= pd.Timestamp(at)]
|
||||||
|
hist_window = hist_window or max(window * 5, 750)
|
||||||
|
out = pd.DataFrame(index=r.columns)
|
||||||
|
roll_sd = r.rolling(window, min_periods=window // 2).std(ddof=1).iloc[-1]
|
||||||
|
roll_mu = r.rolling(window, min_periods=window // 2).mean().iloc[-1]
|
||||||
|
ewm_sd = est.ewm_vol(r, lam).iloc[-1]
|
||||||
|
ewm_mu = est.ewm_mean(r, lam).iloc[-1]
|
||||||
|
out["mean_rolling_ann"] = roll_mu * ppy
|
||||||
|
out["vol_rolling_ann"] = roll_sd * np.sqrt(ppy)
|
||||||
|
out["vol_ewm_ann"] = ewm_sd * np.sqrt(ppy)
|
||||||
|
out["horizon_periods"] = h
|
||||||
|
hist = horizon_returns(r.iloc[-hist_window:], h, overlapping=True)
|
||||||
|
var_fn, cvar_fn = (normal_var, normal_cvar) if space == "log" else (normal_value_var, normal_value_cvar)
|
||||||
|
if drift == "zero":
|
||||||
|
roll_mu, ewm_mu = roll_mu * 0.0, ewm_mu * 0.0
|
||||||
|
hist_sample = hist if space == "log" else np.expm1(hist) # monotone: the same tail, in value units
|
||||||
|
for beta in confidence:
|
||||||
|
cc = f"{int(round(beta * 100)):02d}"
|
||||||
|
out[f"var{cc}_normal_rolling"] = var_fn(roll_mu * h, roll_sd * np.sqrt(h), beta)
|
||||||
|
out[f"cvar{cc}_normal_rolling"] = cvar_fn(roll_mu * h, roll_sd * np.sqrt(h), beta)
|
||||||
|
out[f"var{cc}_normal_ewm"] = var_fn(ewm_mu * h, ewm_sd * np.sqrt(h), beta)
|
||||||
|
out[f"cvar{cc}_normal_ewm"] = cvar_fn(ewm_mu * h, ewm_sd * np.sqrt(h), beta)
|
||||||
|
out[f"var{cc}_hist"] = historical_var(hist_sample, beta)
|
||||||
|
out[f"cvar{cc}_hist"] = historical_cvar(hist_sample, beta)
|
||||||
|
out["n_obs"] = r.notna().sum()
|
||||||
|
out["as_of"] = r.index[-1]
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def risk_series(
|
||||||
|
r: pd.Series, freq: str = "daily", window: int = 200, lam: float = 0.94, confidence=DEFAULT_CONFIDENCE,
|
||||||
|
horizon: str | int = "day",
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Time series of rolling / EWM volatility and Gaussian VaR/CVaR of one return series.
|
||||||
|
|
||||||
|
Historical VaR/CVaR use rolling quantiles / tail means over ``window`` (1-period horizon,
|
||||||
|
scaled with sqrt(h) as an approximation for longer horizons).
|
||||||
|
"""
|
||||||
|
ppy = periods_per_year(freq)
|
||||||
|
h = horizon_periods(horizon, freq)
|
||||||
|
df = pd.DataFrame(index=r.index)
|
||||||
|
roll_sd = est.rolling_vol(r, window)
|
||||||
|
roll_mu = est.rolling_mean(r, window)
|
||||||
|
e_sd = est.ewm_vol(r, lam)
|
||||||
|
e_mu = est.ewm_mean(r, lam)
|
||||||
|
df["vol_rolling_ann"] = roll_sd * np.sqrt(ppy)
|
||||||
|
df["vol_ewm_ann"] = e_sd * np.sqrt(ppy)
|
||||||
|
for beta in confidence:
|
||||||
|
cc = f"{int(round(beta * 100)):02d}"
|
||||||
|
df[f"var{cc}_normal_rolling"] = normal_var(roll_mu * h, roll_sd * np.sqrt(h), beta)
|
||||||
|
df[f"cvar{cc}_normal_rolling"] = normal_cvar(roll_mu * h, roll_sd * np.sqrt(h), beta)
|
||||||
|
df[f"var{cc}_normal_ewm"] = normal_var(e_mu * h, e_sd * np.sqrt(h), beta)
|
||||||
|
df[f"cvar{cc}_normal_ewm"] = normal_cvar(e_mu * h, e_sd * np.sqrt(h), beta)
|
||||||
|
q = est.rolling_quantile(r, window, 1.0 - beta)
|
||||||
|
df[f"var{cc}_hist"] = -q * np.sqrt(h)
|
||||||
|
tail_mean = r.rolling(window, min_periods=window // 2).apply(
|
||||||
|
lambda x, b=beta: -x[x <= np.quantile(x, 1.0 - b)].mean(), raw=True
|
||||||
|
)
|
||||||
|
df[f"cvar{cc}_hist"] = tail_mean * np.sqrt(h)
|
||||||
|
return df
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# portfolio risk
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
@dataclass
|
||||||
|
class PortfolioRisk:
|
||||||
|
"""Risk of a share-based portfolio over one horizon, in return space and in money."""
|
||||||
|
|
||||||
|
as_of: str
|
||||||
|
value: float
|
||||||
|
horizon: str | int
|
||||||
|
horizon_periods: int
|
||||||
|
freq: str
|
||||||
|
method: str
|
||||||
|
confidence: dict # {"0.95": {"var": .., "cvar": .., "var_money": .., "cvar_money": ..}, ...}
|
||||||
|
volatility_ann: float
|
||||||
|
expected_return_horizon: float
|
||||||
|
weights: dict
|
||||||
|
n_obs: int
|
||||||
|
covariance_model: dict
|
||||||
|
|
||||||
|
def as_dict(self) -> dict:
|
||||||
|
return self.__dict__.copy()
|
||||||
|
|
||||||
|
|
||||||
|
def portfolio_risk_from_returns(
|
||||||
|
r: pd.DataFrame, weights: pd.Series, value: float, freq: str, horizon, confidence, method: str,
|
||||||
|
window: int, lam: float, cov_method: str = "rolling", shrinkage: float = 0.0, scenarios: np.ndarray | None = None,
|
||||||
|
) -> PortfolioRisk:
|
||||||
|
"""Portfolio VaR/CVaR from asset returns and weights (fraction of ``value`` per asset).
|
||||||
|
|
||||||
|
``method``: ``'normal'`` (Gaussian with the rolling/EWM covariance), ``'historical'``
|
||||||
|
(realised portfolio returns over the last ``window*5`` periods, aggregated to the horizon)
|
||||||
|
or ``'scenarios'`` (a ``S x N`` array of simulated horizon log-returns, e.g. from
|
||||||
|
:mod:`quantcore.models.simulation`).
|
||||||
|
"""
|
||||||
|
ppy = periods_per_year(freq)
|
||||||
|
h = horizon_periods(horizon, freq)
|
||||||
|
w = weights.reindex(r.columns).fillna(0.0)
|
||||||
|
cov = est.covariance(r, cov_method, window=window, lam=lam, shrinkage=shrinkage)
|
||||||
|
var_p = float(w.values @ cov.values @ w.values)
|
||||||
|
mu_p = float(w.values @ (r.iloc[-window:].mean().fillna(0.0).values))
|
||||||
|
sd_h, mu_h = np.sqrt(var_p * h), mu_p * h
|
||||||
|
levels: dict = {}
|
||||||
|
if method == "normal":
|
||||||
|
for beta in confidence:
|
||||||
|
var_r, cvar_r = normal_var(mu_h, sd_h, beta), normal_cvar(mu_h, sd_h, beta)
|
||||||
|
levels[f"{beta:.2f}"] = _level(var_r, cvar_r, value)
|
||||||
|
n_obs = int(r.iloc[-window:].dropna().shape[0])
|
||||||
|
elif method == "historical":
|
||||||
|
rp = portfolio_log_returns(r.iloc[-max(window * 5, 750):], w.values)
|
||||||
|
rp_h = horizon_returns(rp, h, overlapping=True)
|
||||||
|
for beta in confidence:
|
||||||
|
levels[f"{beta:.2f}"] = _level(float(historical_var(rp_h, beta)), float(historical_cvar(rp_h, beta)), value)
|
||||||
|
n_obs = int(len(rp_h))
|
||||||
|
elif method == "scenarios":
|
||||||
|
if scenarios is None:
|
||||||
|
raise ValueError("method='scenarios' needs a scenario array")
|
||||||
|
pnl = value * (np.expm1(scenarios) @ w.values) # money P&L per scenario
|
||||||
|
for beta in confidence:
|
||||||
|
var_m, cvar_m = scenario_cvar(-pnl, beta)
|
||||||
|
levels[f"{beta:.2f}"] = {
|
||||||
|
"var": float(-np.log1p(-var_m / value)) if var_m < value else np.inf,
|
||||||
|
"cvar": float(-np.log1p(-cvar_m / value)) if cvar_m < value else np.inf,
|
||||||
|
"var_money": var_m, "cvar_money": cvar_m,
|
||||||
|
}
|
||||||
|
n_obs = int(len(scenarios))
|
||||||
|
else:
|
||||||
|
raise ValueError("method must be 'normal', 'historical' or 'scenarios'")
|
||||||
|
return PortfolioRisk(
|
||||||
|
as_of=str(r.index[-1].date()), value=float(value), horizon=horizon, horizon_periods=h, freq=freq,
|
||||||
|
method=method, confidence=levels, volatility_ann=float(np.sqrt(var_p * ppy)),
|
||||||
|
expected_return_horizon=mu_h, weights=w.round(6).to_dict(), n_obs=n_obs,
|
||||||
|
covariance_model={"method": cov_method, "window": window, "lambda": lam, "shrinkage": shrinkage},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _level(var_r: float, cvar_r: float, value: float) -> dict:
|
||||||
|
return {
|
||||||
|
"var": float(var_r), "cvar": float(cvar_r),
|
||||||
|
"var_money": float(loss_to_money(var_r, value)), "cvar_money": float(loss_to_money(cvar_r, value)),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# service facade
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
class RiskService:
|
||||||
|
"""Deterministic risk measurement on top of :class:`~quantcore.data.market_data.MarketDataService`."""
|
||||||
|
|
||||||
|
def __init__(self, market_data):
|
||||||
|
self.md = market_data
|
||||||
|
|
||||||
|
def asset_risk(self, symbols, freq="daily", currency="base", window=200, lam=0.94,
|
||||||
|
confidence=DEFAULT_CONFIDENCE, horizon="day", start=None, end=None,
|
||||||
|
space: str = "log", drift: str = "estimated") -> pd.DataFrame:
|
||||||
|
r = self.md.returns(symbols, start=start, end=end, currency=currency, freq=freq)
|
||||||
|
return asset_risk_table(r, freq, window, lam, confidence, horizon, space=space, drift=drift)
|
||||||
|
|
||||||
|
def risk_series(self, symbol, freq="daily", currency="base", window=200, lam=0.94,
|
||||||
|
confidence=DEFAULT_CONFIDENCE, horizon="day", start=None, end=None) -> pd.DataFrame:
|
||||||
|
r = self.md.returns([symbol], start=start, end=end, currency=currency, freq=freq)[symbol].dropna()
|
||||||
|
return risk_series(r, freq, window, lam, confidence, horizon)
|
||||||
|
|
||||||
|
def covariance(self, symbols, freq="daily", currency="base", method="rolling", window=200, lam=0.94,
|
||||||
|
shrinkage=0.0, start=None, end=None) -> pd.DataFrame:
|
||||||
|
r = self.md.returns(symbols, start=start, end=end, currency=currency, freq=freq)
|
||||||
|
return est.covariance(r, method, window=window, lam=lam, shrinkage=shrinkage)
|
||||||
|
|
||||||
|
def correlation(self, symbols, freq="daily", currency="base", method="rolling", window=200, lam=0.94,
|
||||||
|
start=None, end=None) -> pd.DataFrame:
|
||||||
|
return est.cov_to_corr(self.covariance(symbols, freq, currency, method, window, lam, 0.0, start, end))
|
||||||
|
|
||||||
|
def portfolio_risk(self, holdings: dict[str, float], cash: float = 0.0, freq="daily", currency="base",
|
||||||
|
horizon="day", confidence=DEFAULT_CONFIDENCE, method="normal", cov_method="rolling",
|
||||||
|
window=200, lam=0.94, shrinkage=0.0, start=None, end=None,
|
||||||
|
scenarios: np.ndarray | None = None) -> PortfolioRisk:
|
||||||
|
"""VaR / CVaR of a portfolio given in **numbers of shares** (plus cash), in money terms."""
|
||||||
|
symbols = list(holdings)
|
||||||
|
px = self.md.latest_prices(symbols, currency=currency)
|
||||||
|
values = pd.Series(holdings, dtype=float) * px
|
||||||
|
total = float(values.sum() + cash)
|
||||||
|
if total <= 0:
|
||||||
|
raise ValueError("portfolio value must be positive")
|
||||||
|
weights = values / total
|
||||||
|
r = self.md.returns(symbols, start=start, end=end, currency=currency, freq=freq)
|
||||||
|
return portfolio_risk_from_returns(r, weights, total, freq, horizon, confidence, method, window, lam,
|
||||||
|
cov_method, shrinkage, scenarios)
|
||||||
10
quantcore/models/__init__.py
Normal file
10
quantcore/models/__init__.py
Normal file
@ -0,0 +1,10 @@
|
|||||||
|
"""Stochastic models: covariance estimation, price-scenario simulation, Black-Litterman returns."""
|
||||||
|
|
||||||
|
from quantcore.models.black_litterman import BlackLittermanModel, BlackLittermanResult, View, view_from_sentiment
|
||||||
|
from quantcore.models.covariance import CovarianceModel
|
||||||
|
from quantcore.models.simulation import EmpiricalInnovationSimulator, GBMSimulator, ScenarioSet
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"BlackLittermanModel", "BlackLittermanResult", "View", "view_from_sentiment", "CovarianceModel",
|
||||||
|
"EmpiricalInnovationSimulator", "GBMSimulator", "ScenarioSet",
|
||||||
|
]
|
||||||
231
quantcore/models/black_litterman.py
Normal file
231
quantcore/models/black_litterman.py
Normal file
@ -0,0 +1,231 @@
|
|||||||
|
"""Black-Litterman expected returns with Idzorek's confidence-based view calibration.
|
||||||
|
|
||||||
|
Model (per period, excess returns over the risk-free rate):
|
||||||
|
|
||||||
|
* **Equilibrium**: the market portfolio (capitalisation weights ``w``) is mean-variance
|
||||||
|
optimal for a risk aversion ``delta`` = (mu_M - r_f) / (w' Sigma w), where ``mu_M`` is the
|
||||||
|
average return of the capitalisation-weighted broad market index. The implied equilibrium
|
||||||
|
excess returns are ``pi = delta * Sigma * w``.
|
||||||
|
* **Views**: ``P mu = Q + e``, ``e ~ N(0, Omega)`` with a diagonal ``Omega``. Each view ``k`` has a
|
||||||
|
*confidence* ``C_k`` in (0, 1] (Idzorek 2005): the view's uncertainty ``omega_k`` is the one
|
||||||
|
for which the posterior portfolio moves the fraction ``C_k`` of the way from the market
|
||||||
|
weights to the weights the view would imply at 100 % confidence. For a single view that
|
||||||
|
calibration has the closed form ``omega_k = (1 - C_k) / C_k * p_k (tau Sigma) p_k'``
|
||||||
|
(Walters 2014), which is what :func:`idzorek_omega` uses -- the tilt-matching minimisation
|
||||||
|
of the paper is available as :func:`idzorek_omega_numeric` for verification.
|
||||||
|
* **Posterior**: ``mu_BL = pi + tau Sigma P' (P tau Sigma P' + Omega)^-1 (Q - P pi)`` and
|
||||||
|
``Sigma_BL = Sigma + tau Sigma - tau Sigma P' (P tau Sigma P' + Omega)^-1 P tau Sigma``.
|
||||||
|
|
||||||
|
Units: the covariance is per period (weekly here), ``q`` of a view is stated as an **annual
|
||||||
|
total** return and converted internally. Results are given per period and annualised.
|
||||||
|
|
||||||
|
Reference: T. Idzorek, *A step-by-step guide to the Black-Litterman model, incorporating
|
||||||
|
user-specified confidence levels*, 2005.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
from scipy.optimize import minimize_scalar
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class View:
|
||||||
|
"""A (possibly relative) view: ``sum_i assets[i] * mu_i = q`` with Idzorek confidence."""
|
||||||
|
|
||||||
|
assets: dict[str, float]
|
||||||
|
q: float # annual total return of the view portfolio (log-return units)
|
||||||
|
confidence: float = 0.5 # (0, 1]
|
||||||
|
label: str = ""
|
||||||
|
source: str = "" # provenance, e.g. the insight id that produced the view
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
if not self.assets:
|
||||||
|
raise ValueError("a view needs at least one asset")
|
||||||
|
if not 0 < self.confidence <= 1:
|
||||||
|
raise ValueError("confidence must be in (0, 1]")
|
||||||
|
|
||||||
|
def row(self, symbols: list[str]) -> np.ndarray:
|
||||||
|
p = np.zeros(len(symbols))
|
||||||
|
index = {s: i for i, s in enumerate(symbols)}
|
||||||
|
for asset, weight in self.assets.items():
|
||||||
|
if asset not in index:
|
||||||
|
raise KeyError(f"view asset {asset!r} is not in the model universe")
|
||||||
|
p[index[asset]] = weight
|
||||||
|
return p
|
||||||
|
|
||||||
|
@property
|
||||||
|
def is_relative(self) -> bool:
|
||||||
|
return abs(sum(self.assets.values())) < 1e-12
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class BlackLittermanResult:
|
||||||
|
symbols: list[str]
|
||||||
|
periods_per_year: int
|
||||||
|
tau: float
|
||||||
|
delta: float
|
||||||
|
market_return_ann: float
|
||||||
|
market_volatility_ann: float
|
||||||
|
risk_free_ann: float
|
||||||
|
market_weights: pd.Series
|
||||||
|
prior: pd.Series # equilibrium excess returns per period
|
||||||
|
posterior: pd.Series # posterior excess returns per period
|
||||||
|
posterior_cov: pd.DataFrame # per period
|
||||||
|
omega: np.ndarray
|
||||||
|
views: list[dict] = field(default_factory=list)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def prior_ann(self) -> pd.Series:
|
||||||
|
return self.prior * self.periods_per_year + self.risk_free_ann
|
||||||
|
|
||||||
|
@property
|
||||||
|
def posterior_ann(self) -> pd.Series:
|
||||||
|
return self.posterior * self.periods_per_year + self.risk_free_ann
|
||||||
|
|
||||||
|
@property
|
||||||
|
def posterior_excess_ann(self) -> pd.Series:
|
||||||
|
return self.posterior * self.periods_per_year
|
||||||
|
|
||||||
|
def expected_return_per_period(self, freq_periods_per_year: int) -> pd.Series:
|
||||||
|
"""Posterior total expected log-return per period of another frequency (e.g. daily)."""
|
||||||
|
return self.posterior_ann / freq_periods_per_year
|
||||||
|
|
||||||
|
def unconstrained_weights(self) -> pd.Series:
|
||||||
|
"""Mean-variance weights ``(delta Sigma_BL)^-1 mu_BL`` implied by the posterior."""
|
||||||
|
w = np.linalg.solve(self.delta * self.posterior_cov.values, self.posterior.values)
|
||||||
|
return pd.Series(w, index=self.symbols)
|
||||||
|
|
||||||
|
def as_dict(self) -> dict:
|
||||||
|
return {
|
||||||
|
"periods_per_year": self.periods_per_year, "tau": self.tau, "delta": self.delta,
|
||||||
|
"market_return_ann": self.market_return_ann, "market_volatility_ann": self.market_volatility_ann,
|
||||||
|
"risk_free_ann": self.risk_free_ann, "omega": self.omega.tolist(), "views": self.views,
|
||||||
|
"prior_ann": self.prior_ann.round(6).to_dict(), "posterior_ann": self.posterior_ann.round(6).to_dict(),
|
||||||
|
"market_weights": self.market_weights.round(6).to_dict(),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def idzorek_omega(p: np.ndarray, tau_sigma: np.ndarray, confidence: float) -> float:
|
||||||
|
"""Closed-form Idzorek uncertainty: ``(1 - C) / C * p (tau Sigma) p'`` (C=1 -> 0)."""
|
||||||
|
c = float(np.clip(confidence, 1e-9, 1.0))
|
||||||
|
base = float(p @ tau_sigma @ p)
|
||||||
|
return (1.0 - c) / c * base
|
||||||
|
|
||||||
|
|
||||||
|
def idzorek_omega_numeric(p: np.ndarray, q: float, pi: np.ndarray, sigma: np.ndarray, tau: float, delta: float,
|
||||||
|
confidence: float) -> float:
|
||||||
|
"""Idzorek's original tilt-matching calibration of one view (for verification).
|
||||||
|
|
||||||
|
Finds ``omega`` minimising ``|| w(omega) - (w_mkt + C * (w_100 - w_mkt)) ||`` where
|
||||||
|
``w(omega) = (delta Sigma)^-1 mu_BL(omega)`` and ``w_100`` is the weight vector at 100 %
|
||||||
|
confidence (``omega = 0``).
|
||||||
|
"""
|
||||||
|
ts = tau * sigma
|
||||||
|
w_mkt = np.linalg.solve(delta * sigma, pi)
|
||||||
|
denom_100 = float(p @ ts @ p)
|
||||||
|
mu_100 = pi + ts @ p * (q - p @ pi) / denom_100
|
||||||
|
w_100 = np.linalg.solve(delta * sigma, mu_100)
|
||||||
|
target = w_mkt + confidence * (w_100 - w_mkt)
|
||||||
|
|
||||||
|
def objective(log_omega: float) -> float:
|
||||||
|
omega = np.exp(log_omega)
|
||||||
|
mu = pi + ts @ p * (q - p @ pi) / (denom_100 + omega)
|
||||||
|
w = np.linalg.solve(delta * sigma, mu)
|
||||||
|
return float(np.sum((w - target) ** 2))
|
||||||
|
|
||||||
|
res = minimize_scalar(objective, bounds=(np.log(denom_100) - 25, np.log(denom_100) + 25), method="bounded",
|
||||||
|
options={"xatol": 1e-10})
|
||||||
|
return float(np.exp(res.x))
|
||||||
|
|
||||||
|
|
||||||
|
class BlackLittermanModel:
|
||||||
|
"""Equilibrium returns from market weights and a broad-market average, plus views."""
|
||||||
|
|
||||||
|
def __init__(self, cov: pd.DataFrame, market_weights: pd.Series, market_return: float, risk_free: float = 0.0,
|
||||||
|
tau: float = 0.05, periods_per_year: int = 52):
|
||||||
|
self.symbols = list(cov.index)
|
||||||
|
self.sigma = cov.values.astype(float)
|
||||||
|
w = market_weights.reindex(self.symbols).fillna(0.0).astype(float)
|
||||||
|
if w.sum() <= 0:
|
||||||
|
raise ValueError("market weights must have a positive sum")
|
||||||
|
self.w = (w / w.sum()).values
|
||||||
|
self.market_return = float(market_return) # per period, total
|
||||||
|
self.risk_free = float(risk_free) # per period
|
||||||
|
self.tau = float(tau)
|
||||||
|
self.ppy = int(periods_per_year)
|
||||||
|
self.market_var = float(self.w @ self.sigma @ self.w)
|
||||||
|
self.delta = (self.market_return - self.risk_free) / self.market_var
|
||||||
|
self.pi = self.delta * self.sigma @ self.w # equilibrium excess returns per period
|
||||||
|
|
||||||
|
# -- helpers ---------------------------------------------------------------------------------
|
||||||
|
@property
|
||||||
|
def prior(self) -> pd.Series:
|
||||||
|
return pd.Series(self.pi, index=self.symbols)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def prior_ann(self) -> pd.Series:
|
||||||
|
return self.prior * self.ppy + self.risk_free * self.ppy
|
||||||
|
|
||||||
|
def volatility_ann(self) -> pd.Series:
|
||||||
|
return pd.Series(np.sqrt(np.diag(self.sigma) * self.ppy), index=self.symbols)
|
||||||
|
|
||||||
|
def _views_to_matrices(self, views: list[View], omega_method: str):
|
||||||
|
k, n = len(views), len(self.symbols)
|
||||||
|
P = np.zeros((k, n))
|
||||||
|
Q = np.zeros(k)
|
||||||
|
omega = np.zeros(k)
|
||||||
|
ts = self.tau * self.sigma
|
||||||
|
rf_ann = self.risk_free * self.ppy
|
||||||
|
for i, v in enumerate(views):
|
||||||
|
P[i] = v.row(self.symbols)
|
||||||
|
# annual total -> per-period excess (the risk-free part cancels for relative views)
|
||||||
|
Q[i] = (v.q - rf_ann * P[i].sum()) / self.ppy
|
||||||
|
if omega_method == "idzorek":
|
||||||
|
omega[i] = idzorek_omega(P[i], ts, v.confidence)
|
||||||
|
elif omega_method == "he_litterman":
|
||||||
|
omega[i] = float(P[i] @ ts @ P[i])
|
||||||
|
else:
|
||||||
|
raise ValueError("omega_method must be 'idzorek' or 'he_litterman'")
|
||||||
|
return P, Q, omega
|
||||||
|
|
||||||
|
# -- posterior -------------------------------------------------------------------------------
|
||||||
|
def posterior(self, views: list[View] | None = None, omega_method: str = "idzorek") -> BlackLittermanResult:
|
||||||
|
views = list(views or [])
|
||||||
|
ts = self.tau * self.sigma
|
||||||
|
if not views:
|
||||||
|
mu, cov_post, omega = self.pi.copy(), self.sigma + ts, np.zeros(0)
|
||||||
|
else:
|
||||||
|
P, Q, omega = self._views_to_matrices(views, omega_method)
|
||||||
|
M = P @ ts @ P.T + np.diag(omega)
|
||||||
|
adj = np.linalg.solve(M, Q - P @ self.pi)
|
||||||
|
mu = self.pi + ts @ P.T @ adj
|
||||||
|
cov_post = self.sigma + ts - ts @ P.T @ np.linalg.solve(M, P @ ts)
|
||||||
|
return BlackLittermanResult(
|
||||||
|
symbols=self.symbols, periods_per_year=self.ppy, tau=self.tau, delta=self.delta,
|
||||||
|
market_return_ann=self.market_return * self.ppy, market_volatility_ann=float(np.sqrt(self.market_var * self.ppy)),
|
||||||
|
risk_free_ann=self.risk_free * self.ppy, market_weights=pd.Series(self.w, index=self.symbols),
|
||||||
|
prior=pd.Series(self.pi, index=self.symbols), posterior=pd.Series(mu, index=self.symbols),
|
||||||
|
posterior_cov=pd.DataFrame(cov_post, index=self.symbols, columns=self.symbols), omega=omega,
|
||||||
|
views=[{"label": v.label, "assets": v.assets, "q_ann": v.q, "confidence": v.confidence, "source": v.source,
|
||||||
|
"omega": float(omega[i])} for i, v in enumerate(views)],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def view_from_sentiment(model: BlackLittermanModel, symbol: str, sentiment: float, confidence: float,
|
||||||
|
strength: float = 0.25, label: str = "", source: str = "") -> View:
|
||||||
|
"""Translate a signed assessment into an absolute view on one stock.
|
||||||
|
|
||||||
|
The AI layer delivers *positive / negative* assessments with a degree of belief; the view
|
||||||
|
return is the equilibrium return shifted by ``sentiment * strength`` annual volatilities:
|
||||||
|
``q = pi_ann + r_f + sentiment * strength * sigma_ann``. ``sentiment`` in [-1, 1],
|
||||||
|
``confidence`` in (0, 1] (Idzorek), ``strength`` is a modelling parameter (0.25 by default).
|
||||||
|
"""
|
||||||
|
if not -1.0 <= sentiment <= 1.0:
|
||||||
|
raise ValueError("sentiment must be in [-1, 1]")
|
||||||
|
q = float(model.prior_ann[symbol] + sentiment * strength * model.volatility_ann()[symbol])
|
||||||
|
return View({symbol: 1.0}, q=q, confidence=confidence, label=label or f"{symbol} sentiment {sentiment:+.2f}",
|
||||||
|
source=source)
|
||||||
145
quantcore/models/covariance.py
Normal file
145
quantcore/models/covariance.py
Normal file
@ -0,0 +1,145 @@
|
|||||||
|
"""Covariance model specification and matrix square roots.
|
||||||
|
|
||||||
|
:class:`CovarianceModel` is the *named* estimator the other services refer to, e.g.
|
||||||
|
``CovarianceModel("rolling", window=200)`` (200-day equally weighted) or
|
||||||
|
``CovarianceModel("ewm", lam=0.99)``; it delegates to :mod:`quantcore.measurement.estimators`.
|
||||||
|
|
||||||
|
Square roots: the **symmetric** root ``Sigma^(1/2) = V diag(sqrt(l)) V'`` (from the
|
||||||
|
eigendecomposition; used for the innovation filter, as in the model specification) and the
|
||||||
|
**Cholesky** factor (faster, order dependent). Both satisfy ``A A' = Sigma``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import asdict, dataclass
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from quantcore.measurement import estimators as est
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class CovarianceModel:
|
||||||
|
method: str = "rolling" # 'rolling' | 'ewm'
|
||||||
|
window: int = 200 # rolling window length (observations)
|
||||||
|
lam: float = 0.94 # EWM decay
|
||||||
|
shrinkage: float | str = "auto" # linear shrinkage towards the diagonal: number in [0, 1] or 'auto'
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
if self.method not in ("rolling", "ewm"):
|
||||||
|
raise ValueError("method must be 'rolling' or 'ewm'")
|
||||||
|
if not 0 < self.lam < 1:
|
||||||
|
raise ValueError("lam must be in (0, 1)")
|
||||||
|
if isinstance(self.shrinkage, str):
|
||||||
|
if self.shrinkage != "auto":
|
||||||
|
raise ValueError("shrinkage must be a number in [0, 1] or 'auto'")
|
||||||
|
elif not 0 <= self.shrinkage <= 1:
|
||||||
|
raise ValueError("shrinkage must be in [0, 1]")
|
||||||
|
|
||||||
|
def estimate(self, r: pd.DataFrame, at=None) -> pd.DataFrame:
|
||||||
|
return est.covariance(r, self.method, window=self.window, lam=self.lam, at=at, shrinkage=self.shrinkage)
|
||||||
|
|
||||||
|
def intensity(self, r: pd.DataFrame, at=None) -> float:
|
||||||
|
"""The shrinkage intensity actually applied for ``r`` (resolves ``'auto'``)."""
|
||||||
|
return est.resolve_shrinkage(self.shrinkage, r, self.method, self.window, self.lam, at)
|
||||||
|
|
||||||
|
def sequence(self, r: pd.DataFrame, warmup: int | None = None):
|
||||||
|
return est.covariance_sequence(r, self.method, window=self.window, lam=self.lam, warmup=warmup,
|
||||||
|
shrinkage=self.shrinkage)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def effective_window(self) -> float:
|
||||||
|
return float(self.window) if self.method == "rolling" else 1.0 / (1.0 - self.lam)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def effective_observations(self) -> int:
|
||||||
|
"""Observations the estimator effectively averages over (``3/(1-lambda)`` for EWM)."""
|
||||||
|
return est.effective_observations(self.method, self.window, self.lam)
|
||||||
|
|
||||||
|
def conditioning(self, r: pd.DataFrame, at=None) -> dict:
|
||||||
|
"""Is this estimator able to support ``N`` assets?
|
||||||
|
|
||||||
|
With fewer effective observations than assets the sample covariance is singular (a
|
||||||
|
portfolio in its null space looks riskless, which an optimiser will happily exploit).
|
||||||
|
Shrinkage towards the diagonal repairs it; this report says by how much. ``ratio`` is the
|
||||||
|
smallest eigenvalue as a fraction of the average variance -- below ~1e-6 the matrix is
|
||||||
|
numerically singular, the automatic shrinkage keeps it above 2 %.
|
||||||
|
"""
|
||||||
|
cov = self.estimate(r, at=at).values
|
||||||
|
vals = np.linalg.eigvalsh(cov)
|
||||||
|
scale = float(np.mean(np.diag(cov))) or 1.0
|
||||||
|
n = cov.shape[0]
|
||||||
|
t_eff = self.effective_observations
|
||||||
|
return {
|
||||||
|
"n_assets": n, "effective_observations": t_eff, "observations": int(len(r.dropna())),
|
||||||
|
"shrinkage_applied": round(self.intensity(r, at), 4),
|
||||||
|
"min_eigenvalue_ratio": float(vals.min() / scale),
|
||||||
|
"condition_number": float(vals.max() / vals.min()) if vals.min() > 0 else float("inf"),
|
||||||
|
"rank_deficient": bool(vals.min() <= 1e-10 * scale),
|
||||||
|
"under_determined": bool(t_eff < n),
|
||||||
|
}
|
||||||
|
|
||||||
|
def describe(self) -> dict:
|
||||||
|
d = asdict(self)
|
||||||
|
d["effective_window"] = round(self.effective_window, 4) # rounded: this dict is part of the scenario key
|
||||||
|
return d
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_dict(cls, d: dict | None) -> "CovarianceModel":
|
||||||
|
if d is None:
|
||||||
|
return cls()
|
||||||
|
if isinstance(d, CovarianceModel):
|
||||||
|
return d
|
||||||
|
return cls(**{k: v for k, v in d.items() if k in ("method", "window", "lam", "shrinkage")})
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# matrix roots
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def _as_array(cov) -> np.ndarray:
|
||||||
|
return cov.values if isinstance(cov, pd.DataFrame) else np.asarray(cov, dtype=float)
|
||||||
|
|
||||||
|
|
||||||
|
def symmetric_sqrt(cov, eps_rel: float = 1e-10) -> np.ndarray:
|
||||||
|
"""``Sigma^(1/2)`` via eigendecomposition; tiny/negative eigenvalues are floored."""
|
||||||
|
a = _as_array(cov)
|
||||||
|
vals, vecs = np.linalg.eigh((a + a.T) / 2)
|
||||||
|
floor = eps_rel * max(vals.max(), 1e-300)
|
||||||
|
vals = np.maximum(vals, floor)
|
||||||
|
return (vecs * np.sqrt(vals)) @ vecs.T
|
||||||
|
|
||||||
|
|
||||||
|
def symmetric_inv_sqrt(cov, eps_rel: float = 1e-10) -> np.ndarray:
|
||||||
|
"""``Sigma^(-1/2)`` (pseudo-inverse for directions with negligible variance)."""
|
||||||
|
a = _as_array(cov)
|
||||||
|
vals, vecs = np.linalg.eigh((a + a.T) / 2)
|
||||||
|
floor = eps_rel * max(vals.max(), 1e-300)
|
||||||
|
inv = np.where(vals > floor, 1.0 / np.sqrt(np.maximum(vals, floor)), 0.0)
|
||||||
|
return (vecs * inv) @ vecs.T
|
||||||
|
|
||||||
|
|
||||||
|
def cholesky_factor(cov, jitter_rel: float = 1e-10) -> np.ndarray:
|
||||||
|
"""Lower-triangular ``L`` with ``L L' = Sigma`` (adds a relative jitter if needed)."""
|
||||||
|
a = _as_array(cov)
|
||||||
|
a = (a + a.T) / 2
|
||||||
|
scale = np.trace(a) / len(a)
|
||||||
|
for k in range(6):
|
||||||
|
try:
|
||||||
|
return np.linalg.cholesky(a + np.eye(len(a)) * scale * jitter_rel * (10**k) * (k > 0))
|
||||||
|
except np.linalg.LinAlgError:
|
||||||
|
continue
|
||||||
|
raise np.linalg.LinAlgError("covariance matrix is not positive definite even after jitter")
|
||||||
|
|
||||||
|
|
||||||
|
def matrix_root(cov, kind: str = "symmetric") -> np.ndarray:
|
||||||
|
if kind == "symmetric":
|
||||||
|
return symmetric_sqrt(cov)
|
||||||
|
if kind == "cholesky":
|
||||||
|
return cholesky_factor(cov)
|
||||||
|
raise ValueError("kind must be 'symmetric' or 'cholesky'")
|
||||||
|
|
||||||
|
|
||||||
|
def is_positive_definite(cov, tol: float = 0.0) -> bool:
|
||||||
|
vals = np.linalg.eigvalsh(_as_array(cov))
|
||||||
|
return bool(vals.min() > tol)
|
||||||
255
quantcore/models/sampling.py
Normal file
255
quantcore/models/sampling.py
Normal file
@ -0,0 +1,255 @@
|
|||||||
|
"""Scenario sampling: quasi-Monte-Carlo draws, moment matching, importance sampling, resampling.
|
||||||
|
|
||||||
|
Why this module exists. The mean-CVaR optimiser is a sample-average approximation: it sees the
|
||||||
|
model only through the scenario set. With a plain Monte-Carlo set of ``S`` scenarios the *sample
|
||||||
|
mean* of every asset is off by ``sigma / sqrt(S)`` -- with ``S = 2000`` and weekly volatilities of
|
||||||
|
3-11 % that is 3-13 % per year, the same size as the expected returns themselves -- and the
|
||||||
|
optimiser maximises that noise. Measured on the real universe (30 stocks, 8 seeds): two seeds gave
|
||||||
|
portfolios differing by 100 % of wealth, an in-sample expected return 26 % too high and a true
|
||||||
|
expected return 10 % below the optimum. The remedies implemented here remove that noise almost
|
||||||
|
entirely:
|
||||||
|
|
||||||
|
* **quasi-Monte-Carlo** (scrambled Sobol' points, fixed seed) instead of pseudo-random draws --
|
||||||
|
``S`` is rounded up to a power of two, the first Sobol' coordinates are mapped to the largest
|
||||||
|
principal components (:func:`pca_root`);
|
||||||
|
* **moment matching** -- the (weighted) sample is centred and whitened so that its first two
|
||||||
|
moments are *exactly* the model's; only the tail shape remains sampled;
|
||||||
|
* **importance sampling** -- an exponential tilt of the sampling distribution along a loss
|
||||||
|
direction (a defensive mixture with the nominal density, so the likelihood ratios stay bounded
|
||||||
|
by ``1/(1-sum(mixture))``) puts a quarter of the scenarios in the tail instead of ``1-beta`` of
|
||||||
|
them; scenarios then carry probabilities ``p_s`` that the Rockafellar-Uryasev rows use instead
|
||||||
|
of ``1/S``. Several shifts can be mixed in one set (e.g. one block at ``z_{0.95}`` and one at
|
||||||
|
``z_{0.99}``), which makes the set usable at any confidence level;
|
||||||
|
* **resampling schemes** for the empirical-innovation model: plain bootstrap, balanced bootstrap
|
||||||
|
and Sobol' draws on the ranks of the market-factor innovation.
|
||||||
|
|
||||||
|
Everything is deterministic given the seed; :func:`scenario_hash` fingerprints a scenario set so
|
||||||
|
that two runs can be verified to have used the same one.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
from scipy.optimize import brentq
|
||||||
|
from scipy.stats import norm, qmc
|
||||||
|
|
||||||
|
SAMPLING_METHODS = ("sobol", "mc", "antithetic", "lhs")
|
||||||
|
RESAMPLING_METHODS = ("sobol", "mc", "balanced")
|
||||||
|
|
||||||
|
|
||||||
|
def next_power_of_two(n: int) -> int:
|
||||||
|
return 1 << max(int(n) - 1, 0).bit_length() if n > 1 else 1
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# base samples
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def standard_normal_sample(n: int, dim: int, method: str = "sobol", seed: int | None = 42) -> np.ndarray:
|
||||||
|
"""``n x dim`` standard-normal draws.
|
||||||
|
|
||||||
|
``'sobol'``: scrambled Sobol' points through the inverse normal CDF (``n`` must be a power of
|
||||||
|
two -- use :func:`next_power_of_two`); ``'lhs'``: Latin hypercube; ``'antithetic'``: ``n/2``
|
||||||
|
pseudo-random draws and their negatives; ``'mc'``: pseudo-random (PCG64).
|
||||||
|
"""
|
||||||
|
if method == "sobol":
|
||||||
|
if n & (n - 1):
|
||||||
|
raise ValueError("Sobol' sampling needs a power-of-two sample size")
|
||||||
|
u = qmc.Sobol(d=dim, scramble=True, seed=seed).random(n)
|
||||||
|
return norm.ppf(np.clip(u, 1e-12, 1 - 1e-12))
|
||||||
|
if method == "lhs":
|
||||||
|
u = qmc.LatinHypercube(d=dim, scramble=True, seed=seed).random(n)
|
||||||
|
return norm.ppf(np.clip(u, 1e-12, 1 - 1e-12))
|
||||||
|
rng = np.random.default_rng(seed)
|
||||||
|
if method == "mc":
|
||||||
|
return rng.standard_normal((n, dim))
|
||||||
|
if method == "antithetic":
|
||||||
|
half = rng.standard_normal(((n + 1) // 2, dim))
|
||||||
|
return np.vstack([half, -half])[:n]
|
||||||
|
raise ValueError(f"sampling must be one of {SAMPLING_METHODS}")
|
||||||
|
|
||||||
|
|
||||||
|
def pca_root(cov: np.ndarray) -> np.ndarray:
|
||||||
|
"""``A = V diag(sqrt(l))`` with eigenvalues descending, so ``A A' = cov`` and the first
|
||||||
|
coordinate of a draw maps to the largest principal component (best use of the first Sobol'
|
||||||
|
dimensions)."""
|
||||||
|
vals, vecs = np.linalg.eigh(np.asarray(cov, dtype=float))
|
||||||
|
order = np.argsort(vals)[::-1]
|
||||||
|
vals = np.maximum(vals[order], 0.0)
|
||||||
|
return vecs[:, order] * np.sqrt(vals)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# moment matching
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def moment_match(z: np.ndarray, probabilities: np.ndarray | None = None, target_variance: float = 1.0) -> np.ndarray:
|
||||||
|
"""Affine transform of the rows of ``z`` so that the (weighted) mean is exactly 0 and the
|
||||||
|
(weighted) covariance exactly ``target_variance * I``. Needs more rows than columns."""
|
||||||
|
z = np.asarray(z, dtype=float)
|
||||||
|
n, d = z.shape
|
||||||
|
if n <= d:
|
||||||
|
raise ValueError(f"moment matching needs more scenarios ({n}) than dimensions ({d})")
|
||||||
|
p = np.full(n, 1.0 / n) if probabilities is None else np.asarray(probabilities, dtype=float) / np.sum(probabilities)
|
||||||
|
zc = z - p @ z
|
||||||
|
cov = (zc * p[:, None]).T @ zc
|
||||||
|
vals, vecs = np.linalg.eigh(cov)
|
||||||
|
vals = np.maximum(vals, 1e-14 * vals.max())
|
||||||
|
inv_root = (vecs / np.sqrt(vals)) @ vecs.T
|
||||||
|
return zc @ inv_root * np.sqrt(target_variance)
|
||||||
|
|
||||||
|
|
||||||
|
def effective_sample_size(probabilities: np.ndarray) -> float:
|
||||||
|
p = np.asarray(probabilities, dtype=float)
|
||||||
|
p = p / p.sum()
|
||||||
|
return float(1.0 / np.sum(p * p))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# importance sampling (exponential tilt along a loss direction)
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def loss_direction(root: np.ndarray, exposure: np.ndarray) -> np.ndarray:
|
||||||
|
"""Unit vector in draw space along which the loss of the portfolio with money exposure
|
||||||
|
``exposure`` (shares x price) increases fastest: ``-(A' v) / ||A' v||`` for ``r = mu + A z``."""
|
||||||
|
a = np.asarray(root, dtype=float).T @ np.asarray(exposure, dtype=float)
|
||||||
|
norm_a = np.linalg.norm(a)
|
||||||
|
if norm_a <= 0:
|
||||||
|
raise ValueError("exposure has no risk - cannot tilt")
|
||||||
|
return -a / norm_a
|
||||||
|
|
||||||
|
|
||||||
|
def _as_blocks(shift, mixture) -> list[tuple[float, float]]:
|
||||||
|
"""``[(shift, mixture), ...]`` from scalars or sequences; validates the total mixture."""
|
||||||
|
shifts = [float(shift)] if np.isscalar(shift) else [float(x) for x in shift]
|
||||||
|
mixtures = [float(mixture)] * len(shifts) if np.isscalar(mixture) else [float(x) for x in mixture]
|
||||||
|
if len(mixtures) != len(shifts):
|
||||||
|
raise ValueError("shift and mixture must have the same length")
|
||||||
|
if not shifts or sum(mixtures) <= 0 or sum(mixtures) >= 1 or min(mixtures) <= 0:
|
||||||
|
raise ValueError("mixtures must be positive and sum to less than 1")
|
||||||
|
return list(zip(shifts, mixtures))
|
||||||
|
|
||||||
|
|
||||||
|
def tilt_normal_sample(z: np.ndarray, direction: np.ndarray, shift, mixture=0.5
|
||||||
|
) -> tuple[np.ndarray, np.ndarray]:
|
||||||
|
"""Defensive-mixture importance sampling for standard-normal draws.
|
||||||
|
|
||||||
|
A share ``mixture`` of the rows is shifted by ``theta = shift * direction`` (the loss of the
|
||||||
|
reference portfolio is then centred at its VaR when ``shift = z_beta``); the rest stay
|
||||||
|
nominal. ``shift`` and ``mixture`` may be sequences, in which case one block of rows is
|
||||||
|
shifted per entry -- a set tilted at ``z_{0.95}`` and ``z_{0.99}`` at once serves both
|
||||||
|
confidence levels. Returns the shifted draws and the self-normalised probabilities
|
||||||
|
``p_s ~ f(z_s) / g(z_s)`` with ``g = (1-sum e_k) f + sum_k e_k f(. - theta_k)``, whose
|
||||||
|
likelihood ratios are bounded by ``1 / (1 - sum e_k)``.
|
||||||
|
"""
|
||||||
|
blocks = _as_blocks(shift, mixture)
|
||||||
|
z = np.array(z, dtype=float, copy=True)
|
||||||
|
n = len(z)
|
||||||
|
direction = np.asarray(direction, dtype=float)
|
||||||
|
thetas = [sh * direction for sh, _ in blocks]
|
||||||
|
start = 0
|
||||||
|
for theta, (_, eps) in zip(thetas, blocks):
|
||||||
|
m = int(round(eps * n))
|
||||||
|
z[start : start + m] += theta
|
||||||
|
start += m
|
||||||
|
density = (1.0 - sum(e for _, e in blocks)) * np.ones(n)
|
||||||
|
for theta, (_, eps) in zip(thetas, blocks):
|
||||||
|
density = density + eps * np.exp(np.clip(z @ theta - theta @ theta / 2.0, -700, 700))
|
||||||
|
weights = 1.0 / density
|
||||||
|
return z, weights / weights.sum()
|
||||||
|
|
||||||
|
|
||||||
|
def tilted_day_probabilities(scores: np.ndarray, target_mean: float) -> tuple[np.ndarray, float]:
|
||||||
|
"""Exponential tilt of the empirical distribution of ``scores`` (one per observation) so that
|
||||||
|
the tilted mean equals ``target_mean``: ``q_t ~ exp(theta * score_t)``. Returns ``(q, theta)``."""
|
||||||
|
s = np.asarray(scores, dtype=float)
|
||||||
|
s0 = s - s.mean()
|
||||||
|
if target_mean <= s.mean():
|
||||||
|
return np.full(len(s), 1.0 / len(s)), 0.0
|
||||||
|
if target_mean >= s.max():
|
||||||
|
raise ValueError("target mean beyond the largest score - tilt not feasible")
|
||||||
|
|
||||||
|
def tilted_mean(theta):
|
||||||
|
w = np.exp(theta * (s0 - s0.max()))
|
||||||
|
return float((w * s).sum() / w.sum()) - target_mean
|
||||||
|
|
||||||
|
hi = 1.0
|
||||||
|
while tilted_mean(hi) < 0:
|
||||||
|
hi *= 2.0
|
||||||
|
if hi > 1e6:
|
||||||
|
raise ValueError("tilt did not converge")
|
||||||
|
theta = brentq(tilted_mean, 0.0, hi, xtol=1e-10)
|
||||||
|
w = np.exp(theta * (s0 - s0.max()))
|
||||||
|
return w / w.sum(), float(theta)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# resampling of historical innovations
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def resample_indices(n_obs: int, n_scenarios: int, horizon: int, method: str = "sobol", seed: int | None = 42,
|
||||||
|
order: np.ndarray | None = None) -> np.ndarray:
|
||||||
|
"""``n_scenarios x horizon`` observation indices.
|
||||||
|
|
||||||
|
``'mc'``: uniform bootstrap; ``'balanced'``: every observation used equally often (random
|
||||||
|
permutation of the tiled index set); ``'sobol'``: scrambled Sobol' points in ``horizon``
|
||||||
|
dimensions mapped to ranks -- with ``order`` the observations sorted by, e.g., the market-factor
|
||||||
|
innovation, so that every day of the horizon covers the loss distribution evenly.
|
||||||
|
"""
|
||||||
|
S, h = int(n_scenarios), int(horizon)
|
||||||
|
if method == "mc":
|
||||||
|
return np.random.default_rng(seed).integers(0, n_obs, size=(S, h))
|
||||||
|
if method == "balanced":
|
||||||
|
rng = np.random.default_rng(seed)
|
||||||
|
full, rem = divmod(S * h, n_obs)
|
||||||
|
pool = [rng.permutation(n_obs) for _ in range(full)] + [rng.permutation(n_obs)[:rem]]
|
||||||
|
return rng.permutation(np.concatenate(pool)).reshape(S, h) # every observation full or full + 1 times
|
||||||
|
if method == "sobol":
|
||||||
|
if S & (S - 1):
|
||||||
|
raise ValueError("Sobol' resampling needs a power-of-two sample size")
|
||||||
|
u = qmc.Sobol(d=h, scramble=True, seed=seed).random(S)
|
||||||
|
ranks = np.minimum((u * n_obs).astype(int), n_obs - 1)
|
||||||
|
return ranks if order is None else np.asarray(order)[ranks]
|
||||||
|
raise ValueError(f"resampling must be one of {RESAMPLING_METHODS}")
|
||||||
|
|
||||||
|
|
||||||
|
def tilted_resample(scores: np.ndarray, n_scenarios: int, horizon: int, target_mean, seed: int | None = 42,
|
||||||
|
mixture=0.5) -> tuple[np.ndarray, np.ndarray, list[float]]:
|
||||||
|
"""Importance-sampled bootstrap: one block of scenarios per ``target_mean`` draws its
|
||||||
|
``horizon`` observations from the correspondingly tilted day distribution ``q``
|
||||||
|
(:func:`tilted_day_probabilities` on ``scores`` = loss-direction innovations), the remaining
|
||||||
|
scenarios uniformly. ``target_mean`` and ``mixture`` may be sequences (one block each).
|
||||||
|
Returns ``(indices, probabilities, thetas)`` with self-normalised likelihood ratios
|
||||||
|
``1 / ((1-sum e_k) + sum_k e_k prod_j T q^k_{t_j})``."""
|
||||||
|
blocks = _as_blocks(target_mean, mixture)
|
||||||
|
T = len(scores)
|
||||||
|
rng = np.random.default_rng(seed)
|
||||||
|
S, h = int(n_scenarios), int(horizon)
|
||||||
|
qs, thetas = [], []
|
||||||
|
for target, _ in blocks:
|
||||||
|
q, theta = tilted_day_probabilities(scores, target)
|
||||||
|
qs.append(q)
|
||||||
|
thetas.append(theta)
|
||||||
|
idx = np.empty((S, h), dtype=int)
|
||||||
|
start = 0
|
||||||
|
for q, (_, eps) in zip(qs, blocks):
|
||||||
|
m = int(round(eps * S))
|
||||||
|
idx[start : start + m] = rng.choice(T, size=(m, h), p=q)
|
||||||
|
start += m
|
||||||
|
idx[start:] = rng.integers(0, T, size=(S - start, h))
|
||||||
|
density = (1.0 - sum(e for _, e in blocks)) * np.ones(S)
|
||||||
|
for q, (_, eps) in zip(qs, blocks):
|
||||||
|
density = density + eps * np.exp(np.clip(np.log(T * q[idx]).sum(axis=1), -700, 700))
|
||||||
|
weights = 1.0 / density
|
||||||
|
return idx, weights / weights.sum(), thetas
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# fingerprint
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def scenario_hash(scenarios: np.ndarray, probabilities: np.ndarray | None = None) -> str:
|
||||||
|
"""SHA-256 of the scenario matrix (rounded to 1e-12) and its probabilities -- two optimisations
|
||||||
|
with the same hash used identical inputs."""
|
||||||
|
h = hashlib.sha256()
|
||||||
|
h.update(np.ascontiguousarray(np.round(np.asarray(scenarios, dtype=float), 12)).tobytes())
|
||||||
|
if probabilities is not None:
|
||||||
|
h.update(np.ascontiguousarray(np.round(np.asarray(probabilities, dtype=float), 15)).tobytes())
|
||||||
|
return h.hexdigest()[:16]
|
||||||
403
quantcore/models/simulation.py
Normal file
403
quantcore/models/simulation.py
Normal file
@ -0,0 +1,403 @@
|
|||||||
|
"""Stock-price models: forward simulation of horizon log-returns.
|
||||||
|
|
||||||
|
Both models assume i.i.d. (no autocorrelation) daily or weekly log-returns, so an ``h``-period
|
||||||
|
return is the sum of ``h`` one-period draws.
|
||||||
|
|
||||||
|
**Geometric Brownian motion** (multivariate normal)::
|
||||||
|
|
||||||
|
r_h ~ N(h * m, h * Sigma_T) Sigma_T from the chosen covariance model
|
||||||
|
|
||||||
|
**Empirical innovations** (filtered historical simulation)::
|
||||||
|
|
||||||
|
eps_t = Sigma_{t-1}^(-1/2) r_t standardised with the covariance known at t-1
|
||||||
|
r(j | T) = h * m + Sigma_T^(1/2) sum_{k=1..h} eps_{j_k}, j_k drawn at random from history
|
||||||
|
|
||||||
|
The innovations keep the empirical shape of the return distribution (fat tails, skew,
|
||||||
|
co-movements beyond correlation) while the scale comes from the *latest* covariance.
|
||||||
|
|
||||||
|
**Drift convention.** The expected-return model (Black-Litterman) delivers *arithmetic* expected
|
||||||
|
returns ``mu`` (mean-variance and reverse optimisation are statements about simple returns). The
|
||||||
|
log-drift is therefore ``m = ln(1 + mu_h) / h - diag(Sigma) / 2`` (Ito correction), so that the
|
||||||
|
expected simple return over the horizon is exactly ``mu_h = mu * h``; with the ``'log'`` convention
|
||||||
|
``mu`` is used as the log-drift directly and the expected simple return carries the ``sigma^2/2``
|
||||||
|
bonus (up to +30 % p.a. for the most volatile stocks -- not what a mean-variance model means).
|
||||||
|
|
||||||
|
**Sampling.** Draws come from :mod:`quantcore.models.sampling`: scrambled Sobol' points by
|
||||||
|
default, moment matching so that the scenario mean and covariance are exactly the model's, an
|
||||||
|
optional importance-sampling tilt along the loss direction of a reference portfolio (scenarios
|
||||||
|
then carry probabilities), and a fingerprint of the final set.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from dataclasses import dataclass, field, replace
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
from scipy.stats import norm
|
||||||
|
|
||||||
|
from quantcore.measurement.risk import scenario_cvar, weighted_quantile
|
||||||
|
from quantcore.models import sampling as smp
|
||||||
|
from quantcore.models.covariance import CovarianceModel, matrix_root, symmetric_inv_sqrt
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
DRIFT_CONVENTIONS = ("arithmetic", "log")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class TiltSpec:
|
||||||
|
"""Importance-sampling tilt along the loss direction of the portfolio with money ``exposure``
|
||||||
|
(shares x price per asset).
|
||||||
|
|
||||||
|
``shift`` is given in standard deviations of that portfolio's loss: a block of ``mixture``
|
||||||
|
scenarios is moved by that much, so a shift of ``z_beta`` centres the loss at its VaR. Both
|
||||||
|
may be sequences -- the default tilts one block at ``z_{0.95}`` and one at ``z_{0.99}``, which
|
||||||
|
makes one scenario set serve both confidence levels, so that the set does not depend on the
|
||||||
|
``confidence`` of a particular optimisation run.
|
||||||
|
"""
|
||||||
|
|
||||||
|
exposure: np.ndarray
|
||||||
|
shift: tuple[float, ...] = (1.6448536269514722, 2.3263478740408408) # z_0.95, z_0.99
|
||||||
|
mixture: tuple[float, ...] = (0.25, 0.25)
|
||||||
|
label: str = "market" # provenance: which portfolio defined the direction
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def at_confidence(cls, exposure, confidence: float, mixture: float = 0.5, label: str = "adaptive") -> "TiltSpec":
|
||||||
|
"""A single block shifted to the VaR of ``confidence`` (depends on that confidence)."""
|
||||||
|
return cls(exposure=exposure, shift=(float(norm.ppf(confidence)),), mixture=(float(mixture),), label=label)
|
||||||
|
|
||||||
|
def describe(self) -> dict:
|
||||||
|
return {"shift_sd": list(self.shift), "mixture": list(self.mixture), "direction": self.label}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ScenarioSet:
|
||||||
|
"""``S x N`` matrix of simulated horizon log-returns plus its provenance."""
|
||||||
|
|
||||||
|
log_returns: np.ndarray
|
||||||
|
symbols: list[str]
|
||||||
|
horizon_periods: int
|
||||||
|
freq: str
|
||||||
|
model: str
|
||||||
|
covariance_model: dict
|
||||||
|
as_of: str
|
||||||
|
seed: int | None = None
|
||||||
|
drift: np.ndarray | None = None # horizon log-drift per asset
|
||||||
|
expected_simple: np.ndarray | None = None # model expectation of the horizon simple return
|
||||||
|
probabilities: np.ndarray | None = None # None = equiprobable
|
||||||
|
horizon_variance: np.ndarray | None = None # diag of h * Sigma_T (for :meth:`with_drift`)
|
||||||
|
extra: dict = field(default_factory=dict)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def n_scenarios(self) -> int:
|
||||||
|
return int(self.log_returns.shape[0])
|
||||||
|
|
||||||
|
@property
|
||||||
|
def weights(self) -> np.ndarray:
|
||||||
|
if self.probabilities is None:
|
||||||
|
return np.full(self.n_scenarios, 1.0 / self.n_scenarios)
|
||||||
|
return self.probabilities
|
||||||
|
|
||||||
|
@property
|
||||||
|
def hash(self) -> str:
|
||||||
|
"""Fingerprint of the scenario matrix and its probabilities."""
|
||||||
|
return smp.scenario_hash(self.log_returns, self.probabilities)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def shock_hash(self) -> str:
|
||||||
|
"""Fingerprint of the *shocks* only (the drift removed). Two sets with the same shock hash
|
||||||
|
differ at most in the expected-return model, so portfolios optimised on them see identical
|
||||||
|
risk; a changed view or risk-free rate shifts the scenarios but not this hash."""
|
||||||
|
drift = np.zeros(self.log_returns.shape[1]) if self.drift is None else self.drift
|
||||||
|
return smp.scenario_hash(self.log_returns - drift, self.probabilities)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def shocks(self) -> np.ndarray:
|
||||||
|
"""The scenario matrix with the drift removed: the random part of the model."""
|
||||||
|
return self.log_returns if self.drift is None else self.log_returns - self.drift
|
||||||
|
|
||||||
|
def with_drift(self, mu_period, convention: str = "arithmetic") -> "ScenarioSet":
|
||||||
|
"""The same shocks with a different expected-return model.
|
||||||
|
|
||||||
|
Changing views, the risk-free rate or the expected-return model shifts every scenario by a
|
||||||
|
constant per asset; the random part is untouched. This lets the service keep one scenario
|
||||||
|
set for a whole family of runs (and skips the expensive innovation filter).
|
||||||
|
"""
|
||||||
|
if self.horizon_variance is None:
|
||||||
|
raise ValueError("this ScenarioSet has no horizon variance - cannot re-drift")
|
||||||
|
log_drift, expected_simple = horizon_drift(
|
||||||
|
_mu_vector(mu_period, self.symbols), np.diag(self.horizon_variance) / self.horizon_periods,
|
||||||
|
self.horizon_periods, convention)
|
||||||
|
return replace(self, log_returns=self.shocks + log_drift, drift=log_drift,
|
||||||
|
expected_simple=expected_simple, extra={**self.extra, "drift_convention": convention})
|
||||||
|
|
||||||
|
def simple_returns(self) -> np.ndarray:
|
||||||
|
return np.expm1(self.log_returns)
|
||||||
|
|
||||||
|
def mean_simple_returns(self) -> np.ndarray:
|
||||||
|
"""Probability-weighted scenario mean of the simple returns (the sample counterpart of
|
||||||
|
:attr:`expected_simple`)."""
|
||||||
|
return self.weights @ self.simple_returns()
|
||||||
|
|
||||||
|
def summary(self) -> pd.DataFrame:
|
||||||
|
"""Per-asset moments and quantiles, in both spaces.
|
||||||
|
|
||||||
|
The unprefixed columns are **log**-returns (the space the model works in, where an ``h``-period
|
||||||
|
return is a sum of ``h`` draws); the ``_simple`` columns are the same scenarios as price
|
||||||
|
returns, ``e^x - 1``, which is what a position actually gains or loses and cannot fall below
|
||||||
|
-100 %. A quarterly log-return of -2.0 is a price return of -86 %, so the two must never be
|
||||||
|
confused when the horizon is long.
|
||||||
|
"""
|
||||||
|
r = self.log_returns
|
||||||
|
w = self.weights
|
||||||
|
mean = w @ r
|
||||||
|
std = np.sqrt(w @ (r - mean) ** 2)
|
||||||
|
qs = weighted_quantile(r, (0.01, 0.05, 0.50, 0.95), self.probabilities)
|
||||||
|
simple = np.expm1(r)
|
||||||
|
mean_s = w @ simple
|
||||||
|
std_s = np.sqrt(w @ (simple - mean_s) ** 2)
|
||||||
|
qs_s = np.expm1(qs) # a monotone map, so the quantiles carry over exactly
|
||||||
|
return pd.DataFrame(
|
||||||
|
{
|
||||||
|
"mean": mean, "std": std,
|
||||||
|
"q01": qs[0], "q05": qs[1], "q50": qs[2], "q95": qs[3],
|
||||||
|
"min": r.min(axis=0), "max": r.max(axis=0),
|
||||||
|
"mean_simple": mean_s, "std_simple": std_s,
|
||||||
|
"q01_simple": qs_s[0], "q05_simple": qs_s[1], "q50_simple": qs_s[2], "q95_simple": qs_s[3],
|
||||||
|
"min_simple": np.expm1(r.min(axis=0)), "max_simple": np.expm1(r.max(axis=0)),
|
||||||
|
"expected_simple": self.expected_simple if self.expected_simple is not None else np.expm1(mean),
|
||||||
|
},
|
||||||
|
index=self.symbols,
|
||||||
|
)
|
||||||
|
|
||||||
|
def portfolio_pnl(self, exposure) -> np.ndarray:
|
||||||
|
"""Money P&L per scenario of a portfolio held with money ``exposure`` per asset."""
|
||||||
|
return self.simple_returns() @ np.asarray(exposure, dtype=float)
|
||||||
|
|
||||||
|
def distribution(self, exposure, bins: int = 60, confidence=(0.95, 0.99),
|
||||||
|
quantiles=(0.01, 0.05, 0.25, 0.50, 0.75, 0.95, 0.99)) -> dict:
|
||||||
|
"""The horizon P&L distribution of a portfolio: moments, quantiles, VaR/CVaR, histogram.
|
||||||
|
|
||||||
|
Everything is probability-weighted, so an importance-sampled set (whose raw sample is *not*
|
||||||
|
the model distribution) reports the figures of the model.
|
||||||
|
"""
|
||||||
|
pnl = self.portfolio_pnl(exposure)
|
||||||
|
w = self.weights
|
||||||
|
mean = float(w @ pnl)
|
||||||
|
var = float(w @ (pnl - mean) ** 2)
|
||||||
|
sd = float(np.sqrt(max(var, 0.0)))
|
||||||
|
skew = float(w @ (pnl - mean) ** 3 / sd ** 3) if sd > 0 else 0.0
|
||||||
|
kurt = float(w @ (pnl - mean) ** 4 / sd ** 4) if sd > 0 else 0.0
|
||||||
|
mass, edges = np.histogram(pnl, bins=bins, weights=w)
|
||||||
|
risk = {}
|
||||||
|
for beta in confidence:
|
||||||
|
var_m, cvar_m = scenario_cvar(-pnl, float(beta), self.probabilities)
|
||||||
|
risk[f"{float(beta):g}"] = {"var": float(var_m), "cvar": float(cvar_m)}
|
||||||
|
return {
|
||||||
|
"n_scenarios": self.n_scenarios, "weighted": self.probabilities is not None,
|
||||||
|
"effective_sample_size": round(smp.effective_sample_size(w), 1),
|
||||||
|
"expected_pnl": mean, "volatility": sd, "skewness": skew, "kurtosis": kurt,
|
||||||
|
"min": float(pnl.min()), "max": float(pnl.max()),
|
||||||
|
"quantiles": {f"{q:g}": float(v) for q, v in
|
||||||
|
zip(quantiles, weighted_quantile(pnl, quantiles, self.probabilities))},
|
||||||
|
"risk": risk,
|
||||||
|
"histogram": {"edges": edges.tolist(), "mass": mass.tolist()},
|
||||||
|
}
|
||||||
|
|
||||||
|
def describe(self) -> dict:
|
||||||
|
return {
|
||||||
|
"model": self.model, "n_scenarios": self.n_scenarios, "n_assets": len(self.symbols),
|
||||||
|
"horizon_periods": self.horizon_periods, "freq": self.freq, "as_of": self.as_of, "seed": self.seed,
|
||||||
|
"weighted": self.probabilities is not None,
|
||||||
|
"effective_sample_size": round(smp.effective_sample_size(self.weights), 1),
|
||||||
|
"hash": self.hash, "shock_hash": self.shock_hash, "covariance_model": self.covariance_model,
|
||||||
|
**self.extra,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# helpers
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def _mu_vector(mu, symbols) -> np.ndarray:
|
||||||
|
if mu is None:
|
||||||
|
return np.zeros(len(symbols))
|
||||||
|
if isinstance(mu, pd.Series):
|
||||||
|
mu = mu.reindex(symbols).fillna(0.0).values
|
||||||
|
return np.asarray(mu, dtype=float)
|
||||||
|
|
||||||
|
|
||||||
|
def horizon_drift(mu_period, cov_period, horizon_periods: int, convention: str = "arithmetic"
|
||||||
|
) -> tuple[np.ndarray, np.ndarray]:
|
||||||
|
"""``(log_drift, expected_simple)`` over the horizon for per-period expected returns ``mu``.
|
||||||
|
|
||||||
|
``'arithmetic'``: ``mu`` is the expected simple return per period; the log-drift gets the
|
||||||
|
Ito correction so that ``E[exp(r_h) - 1] = h mu`` exactly under normality.
|
||||||
|
``'log'``: ``mu`` is the log-drift; the expected simple return is ``exp(h mu + h sigma^2/2) - 1``.
|
||||||
|
"""
|
||||||
|
if convention not in DRIFT_CONVENTIONS:
|
||||||
|
raise ValueError(f"drift_convention must be one of {DRIFT_CONVENTIONS}")
|
||||||
|
mu = np.asarray(mu_period, dtype=float)
|
||||||
|
var_h = np.diag(np.asarray(cov_period, dtype=float)) * horizon_periods
|
||||||
|
mu_h = mu * horizon_periods
|
||||||
|
if convention == "arithmetic":
|
||||||
|
mu_h = np.maximum(mu_h, -0.99)
|
||||||
|
return np.log1p(mu_h) - 0.5 * var_h, mu_h
|
||||||
|
return mu_h, np.expm1(mu_h + 0.5 * var_h)
|
||||||
|
|
||||||
|
|
||||||
|
def _finalise(z_scaled: np.ndarray, root: np.ndarray, log_drift: np.ndarray) -> np.ndarray:
|
||||||
|
return log_drift + z_scaled @ root.T
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# GBM
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
class GBMSimulator:
|
||||||
|
"""Multivariate normal horizon returns with the latest covariance of ``cov_model``.
|
||||||
|
|
||||||
|
``root``: ``'pca'`` (eigen-root, largest component first -- the right choice for Sobol'
|
||||||
|
points), ``'symmetric'`` or ``'cholesky'``; the distribution is the same for all three.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, cov_model: CovarianceModel | None = None, root: str = "pca"):
|
||||||
|
self.cov_model = cov_model or CovarianceModel()
|
||||||
|
self.root = root
|
||||||
|
|
||||||
|
def _root(self, cov) -> np.ndarray:
|
||||||
|
return smp.pca_root(cov) if self.root == "pca" else matrix_root(cov, self.root)
|
||||||
|
|
||||||
|
def simulate(self, r: pd.DataFrame, n_scenarios: int = 4096, horizon_periods: int = 1, mu=None,
|
||||||
|
seed: int | None = 42, freq: str = "daily", sampling: str = "sobol", moment_match: bool = True,
|
||||||
|
tilt: TiltSpec | None = None, drift_convention: str = "arithmetic") -> ScenarioSet:
|
||||||
|
symbols = list(r.columns)
|
||||||
|
cov = self.cov_model.estimate(r).values
|
||||||
|
a_h = self._root(cov) * np.sqrt(horizon_periods)
|
||||||
|
log_drift, expected_simple = horizon_drift(_mu_vector(mu, symbols), cov, horizon_periods, drift_convention)
|
||||||
|
n_req = int(n_scenarios)
|
||||||
|
S = smp.next_power_of_two(n_req) if sampling == "sobol" else n_req
|
||||||
|
z = smp.standard_normal_sample(S, len(symbols), sampling, seed)
|
||||||
|
probabilities = None
|
||||||
|
info: dict = {"sampling": sampling, "moment_matched": bool(moment_match), "n_requested": n_req,
|
||||||
|
"drift_convention": drift_convention, "root": self.root}
|
||||||
|
if tilt is not None:
|
||||||
|
direction = smp.loss_direction(a_h, tilt.exposure)
|
||||||
|
z, probabilities = smp.tilt_normal_sample(z, direction, tilt.shift, tilt.mixture)
|
||||||
|
info["importance_sampling"] = tilt.describe()
|
||||||
|
if moment_match:
|
||||||
|
z = smp.moment_match(z, probabilities)
|
||||||
|
scen = _finalise(z, a_h, log_drift)
|
||||||
|
desc = self.cov_model.describe()
|
||||||
|
desc["shrinkage_applied"] = self.cov_model.intensity(r)
|
||||||
|
return ScenarioSet(scen, symbols, horizon_periods, freq, "gbm", desc, str(r.index[-1].date()), seed, log_drift,
|
||||||
|
expected_simple, probabilities, np.diag(cov) * horizon_periods, info)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# empirical innovations
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
class EmpiricalInnovationSimulator:
|
||||||
|
"""Bootstrap of standardised historical innovations rescaled with the latest covariance."""
|
||||||
|
|
||||||
|
def __init__(self, cov_model: CovarianceModel | None = None, root: str = "symmetric", center: bool = True,
|
||||||
|
max_history: int | None = None, clip: float | None = 8.0, standardize: str | None = "diagonal"):
|
||||||
|
self.cov_model = cov_model or CovarianceModel()
|
||||||
|
self.root = root
|
||||||
|
self.center = center
|
||||||
|
self.max_history = max_history
|
||||||
|
self.clip = clip # winsorise standardised innovations at +/- clip (a guard, not a model choice)
|
||||||
|
# re-standardisation of the innovation sample before resampling: 'diagonal' (unit variance per
|
||||||
|
# asset), 'full' (whitening with the sample covariance of the innovations, needs T >> N) or None.
|
||||||
|
# A lagged volatility filter leaves the innovations with variance > 1 on average (the forecast
|
||||||
|
# is too low after every shock); rescaling makes the scenario covariance reproduce Sigma_T.
|
||||||
|
if standardize not in (None, "diagonal", "full"):
|
||||||
|
raise ValueError("standardize must be None, 'diagonal' or 'full'")
|
||||||
|
self.standardize = standardize
|
||||||
|
|
||||||
|
def innovations(self, r: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
"""``eps_t = Sigma_{t-1}^(-1/2) r_t`` for every date after the estimator's warm-up."""
|
||||||
|
r = r.dropna()
|
||||||
|
if self.max_history:
|
||||||
|
r = r.iloc[-self.max_history :]
|
||||||
|
rows, dates = [], []
|
||||||
|
for date, cov_prev, ret in self.cov_model.sequence(r):
|
||||||
|
if self.root == "cholesky":
|
||||||
|
from scipy.linalg import solve_triangular
|
||||||
|
|
||||||
|
from quantcore.models.covariance import cholesky_factor
|
||||||
|
|
||||||
|
eps = solve_triangular(cholesky_factor(cov_prev), ret, lower=True)
|
||||||
|
else:
|
||||||
|
eps = symmetric_inv_sqrt(cov_prev) @ ret
|
||||||
|
rows.append(eps)
|
||||||
|
dates.append(date)
|
||||||
|
if not rows:
|
||||||
|
raise ValueError("no innovations - history shorter than the estimator warm-up")
|
||||||
|
eps = np.vstack(rows)
|
||||||
|
if self.clip:
|
||||||
|
n_clipped = int((np.abs(eps) > self.clip).sum())
|
||||||
|
if n_clipped:
|
||||||
|
log.info("innovations: %d values winsorised at +/-%.0f", n_clipped, self.clip)
|
||||||
|
eps = np.clip(eps, -self.clip, self.clip)
|
||||||
|
return pd.DataFrame(eps, index=pd.DatetimeIndex(dates), columns=r.columns)
|
||||||
|
|
||||||
|
def simulate(self, r: pd.DataFrame, n_scenarios: int = 4096, horizon_periods: int = 1, mu=None,
|
||||||
|
seed: int | None = 42, freq: str = "daily", innovations: pd.DataFrame | None = None,
|
||||||
|
sampling: str = "sobol", moment_match: bool = True, tilt: TiltSpec | None = None,
|
||||||
|
drift_convention: str = "arithmetic") -> ScenarioSet:
|
||||||
|
symbols = list(r.columns)
|
||||||
|
eps = innovations if innovations is not None else self.innovations(r)
|
||||||
|
e = eps.values - (eps.values.mean(axis=0) if self.center else 0.0)
|
||||||
|
e, scale_info = self._restandardize(e)
|
||||||
|
cov = self.cov_model.estimate(r).values
|
||||||
|
a = matrix_root(cov, self.root)
|
||||||
|
log_drift, expected_simple = horizon_drift(_mu_vector(mu, symbols), cov, horizon_periods, drift_convention)
|
||||||
|
h = int(horizon_periods)
|
||||||
|
n_req = int(n_scenarios)
|
||||||
|
S = smp.next_power_of_two(n_req) if sampling == "sobol" else n_req
|
||||||
|
# market-factor ordering of the observations (projection on the dominant direction of Sigma_T)
|
||||||
|
w1 = np.linalg.eigh(cov)[1][:, -1]
|
||||||
|
order = np.argsort(e @ (a @ w1))
|
||||||
|
probabilities = None
|
||||||
|
info: dict = {"n_innovations": len(e), "root": self.root, "standardize": self.standardize,
|
||||||
|
"sampling": sampling, "moment_matched": bool(moment_match), "n_requested": n_req,
|
||||||
|
"drift_convention": drift_convention, **scale_info}
|
||||||
|
if tilt is not None:
|
||||||
|
direction = smp.loss_direction(a, tilt.exposure) # in innovation space
|
||||||
|
scores = e @ direction
|
||||||
|
targets = tuple(sh / np.sqrt(h) for sh in tilt.shift) # per-day mean of the tilted score
|
||||||
|
idx, probabilities, thetas = smp.tilted_resample(scores, S, h, targets, seed, tilt.mixture)
|
||||||
|
info["importance_sampling"] = {**tilt.describe(), "theta": [round(t, 4) for t in thetas]}
|
||||||
|
else:
|
||||||
|
idx = smp.resample_indices(len(e), S, h, sampling, seed, order)
|
||||||
|
summed = e[idx].sum(axis=1) # S x N sums of h innovations, covariance ~ h I
|
||||||
|
if moment_match:
|
||||||
|
summed = smp.moment_match(summed, probabilities, target_variance=float(h))
|
||||||
|
scen = _finalise(summed, a, log_drift)
|
||||||
|
desc = self.cov_model.describe()
|
||||||
|
desc["shrinkage_applied"] = self.cov_model.intensity(r)
|
||||||
|
return ScenarioSet(scen, symbols, horizon_periods, freq, "empirical", desc, str(r.index[-1].date()), seed,
|
||||||
|
log_drift, expected_simple, probabilities, np.diag(cov) * horizon_periods, info)
|
||||||
|
|
||||||
|
def _restandardize(self, e: np.ndarray) -> tuple[np.ndarray, dict]:
|
||||||
|
"""Rescale the centred innovation sample to unit variance (diagonal) or identity covariance (full)."""
|
||||||
|
if self.standardize is None:
|
||||||
|
return e, {}
|
||||||
|
var = e.var(axis=0, ddof=1)
|
||||||
|
info = {"innovation_variance_median": float(np.median(var))}
|
||||||
|
if self.standardize == "diagonal":
|
||||||
|
return e / np.sqrt(np.where(var > 0, var, 1.0)), info
|
||||||
|
n_obs, n = e.shape
|
||||||
|
if n_obs < 3 * n:
|
||||||
|
log.warning("full whitening needs T >= 3N (T=%d, N=%d) - using diagonal rescaling", n_obs, n)
|
||||||
|
return e / np.sqrt(np.where(var > 0, var, 1.0)), info
|
||||||
|
return e @ symmetric_inv_sqrt(np.cov(e, rowvar=False)), info
|
||||||
|
|
||||||
|
|
||||||
|
def make_simulator(model: str, cov_model: CovarianceModel | None = None, **kwargs):
|
||||||
|
if model == "gbm":
|
||||||
|
return GBMSimulator(cov_model, **kwargs)
|
||||||
|
if model in ("empirical", "innovations", "fhs"):
|
||||||
|
return EmpiricalInnovationSimulator(cov_model, **kwargs)
|
||||||
|
raise ValueError("model must be 'gbm' or 'empirical'")
|
||||||
17
quantcore/optimization/__init__.py
Normal file
17
quantcore/optimization/__init__.py
Normal file
@ -0,0 +1,17 @@
|
|||||||
|
"""Mean-CVaR portfolio optimisation in integer share numbers (MILP on cuOpt, HiGHS fallback)."""
|
||||||
|
|
||||||
|
from quantcore.optimization.problem import (
|
||||||
|
MeanCVaRProblem,
|
||||||
|
MILPModel,
|
||||||
|
OptimizationConstraints,
|
||||||
|
ProblemSpec,
|
||||||
|
TransactionCosts,
|
||||||
|
)
|
||||||
|
from quantcore.optimization.service import OptimizationRequest, OptimizationResult, OptimizationService
|
||||||
|
from quantcore.optimization.solvers import CuOptSolver, HighsSolver, SolverResult, solve_with_fallback
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"MeanCVaRProblem", "MILPModel", "OptimizationConstraints", "ProblemSpec", "TransactionCosts",
|
||||||
|
"OptimizationRequest", "OptimizationResult", "OptimizationService", "CuOptSolver", "HighsSolver",
|
||||||
|
"SolverResult", "solve_with_fallback",
|
||||||
|
]
|
||||||
95
quantcore/optimization/gaussian_exact.py
Normal file
95
quantcore/optimization/gaussian_exact.py
Normal file
@ -0,0 +1,95 @@
|
|||||||
|
"""Exact mean-CVaR optimisation under the Gaussian model by outer approximation.
|
||||||
|
|
||||||
|
If the horizon returns are normal, ``R ~ N(m, C)``, the money P&L of ``v = P * n`` is normal and
|
||||||
|
|
||||||
|
CVaR_beta(n) = TC(n) - m'v + k_beta * sqrt(v' C v), k_beta = phi(z_beta) / (1 - beta).
|
||||||
|
|
||||||
|
This is the mean-variance problem in disguise (for elliptical returns every portfolio on the
|
||||||
|
mean-CVaR frontier is on the mean-variance frontier), but in integer shares it is a convex
|
||||||
|
mixed-integer second-order-cone programme, which neither cuOpt nor HiGHS solves directly. The
|
||||||
|
convex function is instead outer-approximated by tangent cuts
|
||||||
|
|
||||||
|
t >= TC(n) + (k_beta C v_k / sigma_k - m)' v (sigma_k = sqrt(v_k' C v_k))
|
||||||
|
|
||||||
|
on the same MILP as the scenario problem (:class:`MeanCVaRProblem` in Gaussian mode). Starting
|
||||||
|
from a cut at a reference portfolio, the MILP is solved, the true CVaR of its solution is
|
||||||
|
compared with the epigraph value ``t``, and a new cut is added at the solution until they agree.
|
||||||
|
Because the integer feasible set is finite and each cut removes the current infeasible point,
|
||||||
|
the loop terminates at the exact optimum (within the solver's MIP gap); measured: 20-70 cuts,
|
||||||
|
identical to Gurobi's direct MISOCP solution to the cent. No scenarios, no sampling error, fully
|
||||||
|
deterministic.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
from collections.abc import Callable
|
||||||
|
from dataclasses import dataclass, field, replace
|
||||||
|
|
||||||
|
from quantcore.optimization.problem import MeanCVaRProblem, MILPModel, ProblemSpec
|
||||||
|
from quantcore.optimization.rowgen import reference_exposure
|
||||||
|
from quantcore.optimization.solvers import SolverResult
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
SolveFn = Callable[[MILPModel], SolverResult]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class OuterApproximationInfo:
|
||||||
|
iterations: int = 0
|
||||||
|
cuts: int = 0
|
||||||
|
gaps: list[float] = field(default_factory=list) # true CVaR - epigraph value per iteration
|
||||||
|
solve_times: list[float] = field(default_factory=list)
|
||||||
|
total_time: float = 0.0
|
||||||
|
converged: bool = False
|
||||||
|
|
||||||
|
def as_dict(self) -> dict:
|
||||||
|
return {"iterations": self.iterations, "cuts": self.cuts, "gaps": [round(g, 4) for g in self.gaps],
|
||||||
|
"solve_times_s": [round(t, 3) for t in self.solve_times], "total_time_s": round(self.total_time, 3),
|
||||||
|
"converged": self.converged}
|
||||||
|
|
||||||
|
|
||||||
|
def solve_gaussian_exact(spec: ProblemSpec, solve: SolveFn, tol: float = 1e-6, max_iter: int = 200
|
||||||
|
) -> tuple[SolverResult, MeanCVaRProblem, OuterApproximationInfo]:
|
||||||
|
"""Outer-approximation loop for a Gaussian-mode :class:`ProblemSpec`.
|
||||||
|
|
||||||
|
``tol`` is relative to the CVaR scale (the limit, or the CVaR itself). Returns the last
|
||||||
|
solver result (its ``x`` decodes with the returned problem), the problem with all cuts, and
|
||||||
|
the iteration log.
|
||||||
|
"""
|
||||||
|
if not spec.is_gaussian:
|
||||||
|
raise ValueError("solve_gaussian_exact needs gaussian_mean and gaussian_cov")
|
||||||
|
t0 = time.monotonic()
|
||||||
|
info = OuterApproximationInfo()
|
||||||
|
problem = MeanCVaRProblem(spec)
|
||||||
|
problem.add_cut(reference_exposure(spec))
|
||||||
|
result: SolverResult | None = None
|
||||||
|
for it in range(1, max_iter + 1):
|
||||||
|
t1 = time.monotonic()
|
||||||
|
result = solve(problem.build())
|
||||||
|
info.solve_times.append(time.monotonic() - t1)
|
||||||
|
info.iterations = it
|
||||||
|
info.cuts = len(problem.cuts)
|
||||||
|
if not result.ok:
|
||||||
|
log.warning("outer approximation: solver returned %s at iteration %d", result.status, it)
|
||||||
|
break
|
||||||
|
d = problem.decode(result.x)
|
||||||
|
gap = d["cvar"] - d["cvar_model"] # true CVaR of the solution minus the epigraph value
|
||||||
|
info.gaps.append(float(gap))
|
||||||
|
scale = max(abs(spec.cvar_limit or 0.0), abs(d["cvar"]), spec.wealth * 1e-4)
|
||||||
|
if gap <= tol * scale:
|
||||||
|
info.converged = True
|
||||||
|
result = replace(result, message=(result.message + f" (outer approximation: {it} iterations, "
|
||||||
|
f"{len(problem.cuts)} cuts)").strip())
|
||||||
|
break
|
||||||
|
if d["sigma"] <= 0: # riskless solution whose cut cannot be formed: the model is exact there
|
||||||
|
info.converged = True
|
||||||
|
break
|
||||||
|
problem.add_cut(d["value"])
|
||||||
|
else:
|
||||||
|
log.warning("outer approximation did not converge in %d iterations", max_iter)
|
||||||
|
result = replace(result, status="FeasibleFound", message=(result.message + " (outer approximation: iteration limit)").strip())
|
||||||
|
info.total_time = time.monotonic() - t0
|
||||||
|
return result, problem, info
|
||||||
544
quantcore/optimization/problem.py
Normal file
544
quantcore/optimization/problem.py
Normal file
@ -0,0 +1,544 @@
|
|||||||
|
"""Mean-CVaR optimisation as a mixed-integer linear programme.
|
||||||
|
|
||||||
|
Decision variables (all share numbers are integers -- a stock can only be bought whole):
|
||||||
|
|
||||||
|
* ``n_i`` final number of shares of stock ``i`` (integer, >= 0: no short positions)
|
||||||
|
* ``b_i``, ``s_i`` shares bought / sold (integers, rebalancing mode only): ``n_i = n0_i + b_i - s_i``
|
||||||
|
* ``alpha`` the VaR level (free) and ``u_s >= 0`` the scenario excess losses of the
|
||||||
|
Rockafellar-Uryasev linearisation of CVaR (scenario mode), or the epigraph variable ``t``
|
||||||
|
of the CVaR (Gaussian mode, see below)
|
||||||
|
* ``y_i`` binary "position open" indicators (only with a cardinality / minimum-position rule)
|
||||||
|
|
||||||
|
**Scenario mode.** With ``S`` scenarios of horizon returns ``R_{s,i}`` (simple returns in the
|
||||||
|
base currency) with probabilities ``p_s`` (``1/S`` unless importance sampling was used), current
|
||||||
|
prices ``P_i`` and transaction costs ``TC``, the loss of scenario ``s`` is
|
||||||
|
``L_s = TC - sum_i P_i R_{s,i} n_i`` and
|
||||||
|
|
||||||
|
CVaR_beta = min_alpha alpha + 1/(1-beta) sum_s p_s max(L_s - alpha, 0)
|
||||||
|
|
||||||
|
which is linear once ``u_s >= L_s - alpha`` is added. The expected P&L in the objective uses the
|
||||||
|
*model* expected returns ``mu_i`` (``sum_i P_i mu_i n_i - TC``) when the spec provides them,
|
||||||
|
otherwise the probability-weighted scenario mean -- the scenarios then only shape the tail.
|
||||||
|
|
||||||
|
**Gaussian mode** (``gaussian_mean``/``gaussian_cov`` given, no scenarios): horizon returns are
|
||||||
|
``R ~ N(m, C)``, the P&L is normal and ``CVaR_beta = TC - m'v + k_beta sqrt(v'Cv)`` with
|
||||||
|
``v = P * n`` and ``k_beta = phi(z_beta)/(1-beta)``. The convex function is represented by an
|
||||||
|
epigraph variable ``t`` and tangent cuts at exposures ``v_k`` (``t >= TC + (k_beta C v_k / sigma_k
|
||||||
|
- m)' v``), added by an outer-approximation loop (:mod:`quantcore.optimization.gaussian_exact`).
|
||||||
|
No scenarios, no sampling error: this is the mean-variance-equivalent exact solution.
|
||||||
|
|
||||||
|
Objectives:
|
||||||
|
|
||||||
|
* ``max_return``: maximise expected P&L subject to ``CVaR_beta <= cvar_limit`` (money);
|
||||||
|
* ``min_cvar``: minimise ``CVaR_beta`` subject to expected P&L ``>= target_pnl``;
|
||||||
|
* ``utility``: maximise ``expected P&L - risk_aversion * CVaR_beta``.
|
||||||
|
|
||||||
|
Constraints: budget (cash cannot go negative), maximum weight per stock and per sector
|
||||||
|
(fractions of total wealth), optional transaction-cost and turnover limits, optional
|
||||||
|
cardinality / minimum position size. Weight limits are tight *variable bounds*
|
||||||
|
(``n_i <= floor(w_max W / P_i)``), which helps the MILP solver considerably; ``alpha`` and the
|
||||||
|
``u_s`` are bounded by the worst gain and loss attainable within those caps (see
|
||||||
|
``_loss_bounds``) -- free variables make the CVaR objective numerically hard.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import scipy.sparse as sp
|
||||||
|
from scipy.stats import norm
|
||||||
|
|
||||||
|
from quantcore.measurement.risk import scenario_cvar
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class OptimizationConstraints:
|
||||||
|
max_weight: float = 0.10 # per stock, fraction of total wealth
|
||||||
|
max_sector_weight: float | dict[str, float] = 0.25 # per GICS sector (scalar = same for all)
|
||||||
|
max_positions: int | None = None # cardinality limit (adds binaries)
|
||||||
|
min_position_value: float = 0.0 # minimum value of an open position (adds binaries)
|
||||||
|
max_turnover: float | None = None # fraction of wealth traded (buys + sells)
|
||||||
|
max_transaction_cost: float | None = None # absolute money amount
|
||||||
|
min_invested: float = 0.0 # minimum fraction of wealth invested in stocks
|
||||||
|
max_invested: float = 1.0 # maximum fraction invested (1.0 = fully, remainder is cash)
|
||||||
|
|
||||||
|
def sector_limit(self, sector: str) -> float | None:
|
||||||
|
if isinstance(self.max_sector_weight, dict):
|
||||||
|
return self.max_sector_weight.get(sector, self.max_sector_weight.get("*"))
|
||||||
|
return self.max_sector_weight
|
||||||
|
|
||||||
|
def uses_binaries(self) -> bool:
|
||||||
|
return self.max_positions is not None or self.min_position_value > 0
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class TransactionCosts:
|
||||||
|
proportional_buy: float = 0.001 # 10 bp of the traded value
|
||||||
|
proportional_sell: float = 0.001
|
||||||
|
fixed_per_trade: float = 0.0 # money per stock traded (adds trade binaries)
|
||||||
|
|
||||||
|
def uses_binaries(self) -> bool:
|
||||||
|
return self.fixed_per_trade > 0
|
||||||
|
|
||||||
|
|
||||||
|
def cvar_multiplier(beta: float) -> float:
|
||||||
|
"""``k_beta = phi(z_beta) / (1 - beta)``: Gaussian CVaR in standard deviations."""
|
||||||
|
return float(norm.pdf(norm.ppf(beta)) / (1.0 - beta))
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ProblemSpec:
|
||||||
|
"""Everything the builder needs, as plain arrays (built by the service, tested directly)."""
|
||||||
|
|
||||||
|
symbols: list[str]
|
||||||
|
prices: np.ndarray # current price per share, base currency
|
||||||
|
scenarios: np.ndarray | None # S x N simple returns over the horizon (None in Gaussian mode)
|
||||||
|
sectors: list[str]
|
||||||
|
cash: float # initial cash
|
||||||
|
holdings: np.ndarray | None = None # initial shares (None or zeros: new portfolio)
|
||||||
|
confidence: float = 0.95
|
||||||
|
cvar_limit: float | None = None # money
|
||||||
|
objective: str = "max_return" # 'max_return' | 'min_cvar' | 'utility'
|
||||||
|
risk_aversion: float = 1.0 # for 'utility'
|
||||||
|
target_pnl: float | None = None # for 'min_cvar' (money over the horizon)
|
||||||
|
constraints: OptimizationConstraints = field(default_factory=OptimizationConstraints)
|
||||||
|
costs: TransactionCosts = field(default_factory=TransactionCosts)
|
||||||
|
probabilities: np.ndarray | None = None # scenario probabilities (None = 1/S); may sum to < 1 for a subset
|
||||||
|
expected_returns: np.ndarray | None = None # model expected simple return per asset over the horizon
|
||||||
|
gaussian_mean: np.ndarray | None = None # Gaussian mode: mean of the horizon simple returns
|
||||||
|
gaussian_cov: np.ndarray | None = None # Gaussian mode: covariance of the horizon simple returns
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
self.prices = np.asarray(self.prices, dtype=float)
|
||||||
|
n = len(self.symbols)
|
||||||
|
if self.prices.shape != (n,):
|
||||||
|
raise ValueError("prices must have one entry per symbol")
|
||||||
|
if len(self.sectors) != n:
|
||||||
|
raise ValueError("sectors must have one entry per symbol")
|
||||||
|
if self.holdings is None:
|
||||||
|
self.holdings = np.zeros(n)
|
||||||
|
self.holdings = np.asarray(self.holdings, dtype=float)
|
||||||
|
if np.any(self.holdings < 0):
|
||||||
|
raise ValueError("initial holdings cannot be negative")
|
||||||
|
if not 0 < self.confidence < 1:
|
||||||
|
raise ValueError("confidence must be in (0, 1)")
|
||||||
|
if self.objective == "max_return" and self.cvar_limit is None:
|
||||||
|
raise ValueError("objective 'max_return' needs a cvar_limit")
|
||||||
|
if self.objective == "min_cvar" and self.target_pnl is None:
|
||||||
|
raise ValueError("objective 'min_cvar' needs a target_pnl")
|
||||||
|
if self.objective not in ("max_return", "min_cvar", "utility"):
|
||||||
|
raise ValueError("objective must be 'max_return', 'min_cvar' or 'utility'")
|
||||||
|
if self.is_gaussian:
|
||||||
|
self.gaussian_mean = np.asarray(self.gaussian_mean, dtype=float)
|
||||||
|
self.gaussian_cov = np.asarray(self.gaussian_cov, dtype=float)
|
||||||
|
if self.gaussian_mean.shape != (n,) or self.gaussian_cov.shape != (n, n):
|
||||||
|
raise ValueError("gaussian_mean must be N and gaussian_cov N x N")
|
||||||
|
self.scenarios = np.zeros((0, n))
|
||||||
|
if self.expected_returns is None:
|
||||||
|
self.expected_returns = self.gaussian_mean
|
||||||
|
else:
|
||||||
|
if self.scenarios is None:
|
||||||
|
raise ValueError("scenarios are required unless gaussian_mean/gaussian_cov are given")
|
||||||
|
self.scenarios = np.asarray(self.scenarios, dtype=float)
|
||||||
|
if self.scenarios.ndim != 2 or self.scenarios.shape[1] != n or self.scenarios.shape[0] == 0:
|
||||||
|
raise ValueError("scenarios must be S x N with S >= 1")
|
||||||
|
if self.probabilities is not None:
|
||||||
|
self.probabilities = np.asarray(self.probabilities, dtype=float)
|
||||||
|
if self.probabilities.shape != (self.scenarios.shape[0],):
|
||||||
|
raise ValueError("probabilities must have one entry per scenario")
|
||||||
|
if np.any(self.probabilities < 0) or self.probabilities.sum() > 1.0 + 1e-9:
|
||||||
|
raise ValueError("probabilities must be non-negative and sum to at most 1")
|
||||||
|
if self.expected_returns is not None:
|
||||||
|
self.expected_returns = np.asarray(self.expected_returns, dtype=float)
|
||||||
|
if self.expected_returns.shape != (n,):
|
||||||
|
raise ValueError("expected_returns must have one entry per symbol")
|
||||||
|
|
||||||
|
@property
|
||||||
|
def is_gaussian(self) -> bool:
|
||||||
|
return self.gaussian_mean is not None and self.gaussian_cov is not None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def n_scenarios(self) -> int:
|
||||||
|
return int(self.scenarios.shape[0])
|
||||||
|
|
||||||
|
@property
|
||||||
|
def weights(self) -> np.ndarray:
|
||||||
|
"""Scenario probabilities (``1/S`` when none were given)."""
|
||||||
|
if self.probabilities is None:
|
||||||
|
return np.full(self.n_scenarios, 1.0 / max(self.n_scenarios, 1))
|
||||||
|
return self.probabilities
|
||||||
|
|
||||||
|
@property
|
||||||
|
def mean_returns(self) -> np.ndarray:
|
||||||
|
"""Expected simple return per asset used in the objective."""
|
||||||
|
if self.expected_returns is not None:
|
||||||
|
return self.expected_returns
|
||||||
|
w = self.weights
|
||||||
|
return (w @ self.scenarios) / w.sum()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def rebalancing(self) -> bool:
|
||||||
|
return bool(np.any(self.holdings > 0))
|
||||||
|
|
||||||
|
@property
|
||||||
|
def wealth(self) -> float:
|
||||||
|
return float(self.cash + self.holdings @ self.prices)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class MILPModel:
|
||||||
|
"""Solver-agnostic MILP: ``min/max c'x s.t. row_lb <= A x <= row_ub, var_lb <= x <= var_ub``."""
|
||||||
|
|
||||||
|
c: np.ndarray
|
||||||
|
A: sp.csr_matrix
|
||||||
|
row_lb: np.ndarray
|
||||||
|
row_ub: np.ndarray
|
||||||
|
var_lb: np.ndarray
|
||||||
|
var_ub: np.ndarray
|
||||||
|
var_types: list[str] # 'I' | 'C' (binaries are 'I' with bounds [0, 1])
|
||||||
|
var_names: list[str]
|
||||||
|
row_names: list[str]
|
||||||
|
maximize: bool
|
||||||
|
offset: float = 0.0
|
||||||
|
|
||||||
|
@property
|
||||||
|
def n_vars(self) -> int:
|
||||||
|
return len(self.c)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def n_rows(self) -> int:
|
||||||
|
return self.A.shape[0]
|
||||||
|
|
||||||
|
@property
|
||||||
|
def nnz(self) -> int:
|
||||||
|
return int(self.A.nnz)
|
||||||
|
|
||||||
|
def describe(self) -> dict:
|
||||||
|
return {"n_vars": self.n_vars, "n_int": sum(t == "I" for t in self.var_types), "n_rows": self.n_rows,
|
||||||
|
"nnz": self.nnz, "maximize": self.maximize}
|
||||||
|
|
||||||
|
|
||||||
|
class MeanCVaRProblem:
|
||||||
|
"""Builds the MILP for a :class:`ProblemSpec` and decodes solver vectors."""
|
||||||
|
|
||||||
|
def __init__(self, spec: ProblemSpec):
|
||||||
|
self.spec = spec
|
||||||
|
self.n = len(spec.symbols)
|
||||||
|
self.gaussian = spec.is_gaussian
|
||||||
|
self.S = 0 if self.gaussian else spec.n_scenarios
|
||||||
|
self.rebalancing = spec.rebalancing
|
||||||
|
self.use_pos_bin = spec.constraints.uses_binaries()
|
||||||
|
self.use_trade_bin = spec.costs.uses_binaries()
|
||||||
|
self.cuts: list[np.ndarray] = [] # Gaussian mode: exposures v_k of the tangent cuts
|
||||||
|
self._layout()
|
||||||
|
|
||||||
|
# -- variable layout -----------------------------------------------------------------------
|
||||||
|
def _layout(self) -> None:
|
||||||
|
n = self.n
|
||||||
|
self.i_n = np.arange(0, n)
|
||||||
|
pos = n
|
||||||
|
if self.rebalancing:
|
||||||
|
self.i_b = np.arange(pos, pos + n)
|
||||||
|
pos += n
|
||||||
|
self.i_s = np.arange(pos, pos + n)
|
||||||
|
pos += n
|
||||||
|
else:
|
||||||
|
self.i_b = self.i_s = None
|
||||||
|
if self.gaussian:
|
||||||
|
self.i_alpha = None
|
||||||
|
self.i_u = np.arange(0)
|
||||||
|
self.i_cv = pos # epigraph variable of the CVaR
|
||||||
|
pos += 1
|
||||||
|
else:
|
||||||
|
self.i_alpha = pos
|
||||||
|
pos += 1
|
||||||
|
self.i_u = np.arange(pos, pos + self.S)
|
||||||
|
pos += self.S
|
||||||
|
self.i_cv = None
|
||||||
|
if self.use_pos_bin:
|
||||||
|
self.i_y = np.arange(pos, pos + n)
|
||||||
|
pos += n
|
||||||
|
else:
|
||||||
|
self.i_y = None
|
||||||
|
if self.use_trade_bin:
|
||||||
|
self.i_t = np.arange(pos, pos + n)
|
||||||
|
pos += n
|
||||||
|
else:
|
||||||
|
self.i_t = None
|
||||||
|
self.n_vars = pos
|
||||||
|
|
||||||
|
# -- helpers --------------------------------------------------------------------------------
|
||||||
|
def _max_shares(self) -> np.ndarray:
|
||||||
|
"""Tight upper bound per stock: ``floor(min(w_max, f_max) * W / P_i)`` (the weight limit, or the
|
||||||
|
invested fraction if that is smaller). A legacy holding above the cap has to be sold down."""
|
||||||
|
w = self.spec.wealth
|
||||||
|
lim = min(self.spec.constraints.max_weight, self.spec.constraints.max_invested) * w
|
||||||
|
ub = np.floor(lim / self.spec.prices + 1e-9)
|
||||||
|
return np.maximum(ub, 0.0)
|
||||||
|
|
||||||
|
def _loss_bounds(self, ub_shares: np.ndarray) -> tuple[float, float]:
|
||||||
|
"""Valid finite bounds ``(max_gain, max_loss)`` over the box ``0 <= n_i <= ub_shares_i``.
|
||||||
|
|
||||||
|
Any feasible portfolio's money P&L lies in ``[-max_loss, max_gain]``, so ``alpha`` and the
|
||||||
|
``u_s`` can be bounded: free variables with infinite bounds make the CVaR objective
|
||||||
|
numerically hard for the solvers (cuOpt's dual simplex aborts on such models).
|
||||||
|
"""
|
||||||
|
p = self.spec.prices
|
||||||
|
cost_ub = float(self.spec.costs.proportional_buy + self.spec.costs.proportional_sell) * float(
|
||||||
|
(ub_shares * p).sum()) + float(self.spec.costs.fixed_per_trade) * self.n
|
||||||
|
if self.gaussian:
|
||||||
|
v_max = ub_shares * p
|
||||||
|
mean = np.abs(self.spec.gaussian_mean) @ v_max
|
||||||
|
sigma = float(np.sqrt(max(v_max @ self.spec.gaussian_cov @ v_max, 0.0)))
|
||||||
|
spread = mean + cvar_multiplier(self.spec.confidence) * sigma + cost_ub
|
||||||
|
return spread, spread
|
||||||
|
gain = self.spec.scenarios * p[None, :] # money gain per share, per scenario
|
||||||
|
max_gain = float(np.maximum(gain.max(axis=0), 0.0) @ ub_shares)
|
||||||
|
max_loss = float(np.maximum(-gain.min(axis=0), 0.0) @ ub_shares) + cost_ub
|
||||||
|
return max_gain, max_loss
|
||||||
|
|
||||||
|
def _cost_terms(self):
|
||||||
|
"""Coefficients of the transaction cost ``TC`` per variable (b, s, t or n)."""
|
||||||
|
p, c = self.spec.prices, self.spec.costs
|
||||||
|
terms = [] # (indices, coefficients)
|
||||||
|
if self.rebalancing:
|
||||||
|
terms.append((self.i_b, c.proportional_buy * p))
|
||||||
|
terms.append((self.i_s, c.proportional_sell * p))
|
||||||
|
else:
|
||||||
|
terms.append((self.i_n, c.proportional_buy * p))
|
||||||
|
if self.use_trade_bin:
|
||||||
|
terms.append((self.i_t, np.full(self.n, c.fixed_per_trade)))
|
||||||
|
return terms
|
||||||
|
|
||||||
|
def _cvar_expr(self) -> tuple[np.ndarray, np.ndarray]:
|
||||||
|
"""Indices and coefficients of the linear expression equal to ``CVaR_beta``."""
|
||||||
|
if self.gaussian:
|
||||||
|
return np.array([self.i_cv]), np.array([1.0])
|
||||||
|
tail = self.spec.weights / (1.0 - self.spec.confidence)
|
||||||
|
return np.concatenate([[self.i_alpha], self.i_u]), np.concatenate([[1.0], tail])
|
||||||
|
|
||||||
|
# -- Gaussian mode -----------------------------------------------------------------------
|
||||||
|
def gaussian_risk(self, exposure: np.ndarray, total_cost: float = 0.0) -> tuple[float, float, float]:
|
||||||
|
"""``(VaR, CVaR, sigma)`` of the money P&L for exposure ``v`` under the Gaussian model."""
|
||||||
|
v = np.asarray(exposure, dtype=float)
|
||||||
|
sigma = float(np.sqrt(max(v @ self.spec.gaussian_cov @ v, 0.0)))
|
||||||
|
mean = float(self.spec.gaussian_mean @ v) - total_cost
|
||||||
|
z = float(norm.ppf(self.spec.confidence))
|
||||||
|
return -mean + z * sigma, -mean + cvar_multiplier(self.spec.confidence) * sigma, sigma
|
||||||
|
|
||||||
|
def add_cut(self, exposure: np.ndarray) -> None:
|
||||||
|
"""Add a tangent cut of the Gaussian CVaR at money exposure ``v_k`` (``v_k`` must carry risk)."""
|
||||||
|
if not self.gaussian:
|
||||||
|
raise ValueError("cuts only exist in Gaussian mode")
|
||||||
|
v = np.asarray(exposure, dtype=float)
|
||||||
|
if v @ self.spec.gaussian_cov @ v <= 0:
|
||||||
|
raise ValueError("cut exposure has no risk")
|
||||||
|
self.cuts.append(v)
|
||||||
|
|
||||||
|
# -- build ------------------------------------------------------------------------------------
|
||||||
|
def build(self) -> MILPModel:
|
||||||
|
sp_ = self.spec
|
||||||
|
n, S = self.n, self.S
|
||||||
|
p = sp_.prices
|
||||||
|
W = sp_.wealth
|
||||||
|
mean_gain = sp_.mean_returns * p # expected money gain per share
|
||||||
|
cost_terms = self._cost_terms()
|
||||||
|
ub_shares = self._max_shares()
|
||||||
|
|
||||||
|
rows, cols, vals = [], [], []
|
||||||
|
row_lb, row_ub, row_names = [], [], []
|
||||||
|
r = 0
|
||||||
|
|
||||||
|
def add_row(idx, coef, lb, ub, name):
|
||||||
|
nonlocal r
|
||||||
|
idx = np.asarray(idx)
|
||||||
|
coef = np.asarray(coef, dtype=float)
|
||||||
|
keep = coef != 0
|
||||||
|
rows.extend([r] * int(keep.sum()))
|
||||||
|
cols.extend(idx[keep].tolist())
|
||||||
|
vals.extend(coef[keep].tolist())
|
||||||
|
row_lb.append(lb)
|
||||||
|
row_ub.append(ub)
|
||||||
|
row_names.append(name)
|
||||||
|
r += 1
|
||||||
|
|
||||||
|
cost_idx = np.concatenate([i for i, _ in cost_terms])
|
||||||
|
cost_coef = np.concatenate([c for _, c in cost_terms])
|
||||||
|
|
||||||
|
if self.gaussian:
|
||||||
|
# 1g. tangent cuts: TC + (k C v_k / sigma_k - m)' (P n) - t <= 0
|
||||||
|
if not self.cuts:
|
||||||
|
raise ValueError("Gaussian mode needs at least one cut (use gaussian_exact.solve)")
|
||||||
|
k = cvar_multiplier(sp_.confidence)
|
||||||
|
for j, v_k in enumerate(self.cuts):
|
||||||
|
sigma_k = np.sqrt(v_k @ sp_.gaussian_cov @ v_k)
|
||||||
|
grad = k * (sp_.gaussian_cov @ v_k) / sigma_k - sp_.gaussian_mean
|
||||||
|
idx = np.concatenate([self.i_n, cost_idx, [self.i_cv]])
|
||||||
|
coef = np.concatenate([grad * p, cost_coef, [-1.0]])
|
||||||
|
add_row(idx, coef, -np.inf, 0.0, f"cut_{j}")
|
||||||
|
else:
|
||||||
|
# 1. CVaR scenario rows: sum_i gain_si n_i - TC + alpha + u_s >= 0
|
||||||
|
gain = sp_.scenarios * p[None, :] # money gain per share in each scenario
|
||||||
|
for s in range(S):
|
||||||
|
idx = np.concatenate([self.i_n, cost_idx, [self.i_alpha, self.i_u[s]]])
|
||||||
|
coef = np.concatenate([gain[s], -cost_coef, [1.0, 1.0]])
|
||||||
|
add_row(idx, coef, 0.0, np.inf, f"cvar_scen_{s}")
|
||||||
|
|
||||||
|
# 2. CVaR expression rows (limit for max_return, objective otherwise)
|
||||||
|
cvar_idx, cvar_coef = self._cvar_expr()
|
||||||
|
if sp_.objective == "max_return" or sp_.cvar_limit is not None:
|
||||||
|
add_row(cvar_idx, cvar_coef, -np.inf, float(sp_.cvar_limit), "cvar_limit")
|
||||||
|
|
||||||
|
# 3. budget: cash_end = cash0 + sum p s - sum p b - TC >= 0 (duplicate indices are summed)
|
||||||
|
if self.rebalancing:
|
||||||
|
add_row(np.concatenate([self.i_b, self.i_s, cost_idx]), np.concatenate([p, -p, cost_coef]), -np.inf,
|
||||||
|
sp_.cash, "budget")
|
||||||
|
# 4. linking n - b + s = n0
|
||||||
|
for i in range(n):
|
||||||
|
add_row([self.i_n[i], self.i_b[i], self.i_s[i]], [1.0, -1.0, 1.0], sp_.holdings[i], sp_.holdings[i],
|
||||||
|
f"link_{i}")
|
||||||
|
else:
|
||||||
|
add_row(np.concatenate([self.i_n, cost_idx]), np.concatenate([p, cost_coef]), -np.inf, sp_.cash, "budget")
|
||||||
|
|
||||||
|
# 5. invested fraction bounds (min / max)
|
||||||
|
cons = sp_.constraints
|
||||||
|
lo = cons.min_invested * W if cons.min_invested > 0 else -np.inf
|
||||||
|
hi = cons.max_invested * W if cons.max_invested < 1.0 else np.inf
|
||||||
|
if np.isfinite(lo) or np.isfinite(hi):
|
||||||
|
add_row(self.i_n, p, lo, hi, "invested")
|
||||||
|
|
||||||
|
# 6. sector limits
|
||||||
|
sectors = np.asarray(sp_.sectors)
|
||||||
|
for sec in sorted(set(sectors)):
|
||||||
|
lim = cons.sector_limit(sec)
|
||||||
|
if lim is None or lim >= cons.max_invested:
|
||||||
|
continue
|
||||||
|
members = np.where(sectors == sec)[0]
|
||||||
|
add_row(self.i_n[members], p[members], -np.inf, lim * W, f"sector_{sec}")
|
||||||
|
|
||||||
|
# 7. transaction cost limit and 8. turnover
|
||||||
|
if cons.max_transaction_cost is not None:
|
||||||
|
add_row(cost_idx, cost_coef, -np.inf, cons.max_transaction_cost, "max_cost")
|
||||||
|
if cons.max_turnover is not None and self.rebalancing:
|
||||||
|
add_row(np.concatenate([self.i_b, self.i_s]), np.concatenate([p, p]), -np.inf, cons.max_turnover * W,
|
||||||
|
"turnover")
|
||||||
|
|
||||||
|
# 9. cardinality / minimum position size via binaries y_i
|
||||||
|
if self.use_pos_bin:
|
||||||
|
for i in range(n):
|
||||||
|
add_row([self.i_n[i], self.i_y[i]], [1.0, -max(ub_shares[i], 1.0)], -np.inf, 0.0, f"open_{i}")
|
||||||
|
if cons.min_position_value > 0:
|
||||||
|
add_row([self.i_n[i], self.i_y[i]], [p[i], -cons.min_position_value], 0.0, np.inf, f"minpos_{i}")
|
||||||
|
if cons.max_positions is not None:
|
||||||
|
add_row(self.i_y, np.ones(n), -np.inf, float(cons.max_positions), "cardinality")
|
||||||
|
|
||||||
|
# 10. trade indicators for fixed costs: (b_i + s_i) <= M t_i (new mode: n_i <= M t_i)
|
||||||
|
if self.use_trade_bin:
|
||||||
|
big_m = np.maximum(ub_shares, sp_.holdings) + 1.0
|
||||||
|
for i in range(n):
|
||||||
|
if self.rebalancing:
|
||||||
|
add_row([self.i_b[i], self.i_s[i], self.i_t[i]], [1.0, 1.0, -big_m[i]], -np.inf, 0.0, f"trade_{i}")
|
||||||
|
else:
|
||||||
|
add_row([self.i_n[i], self.i_t[i]], [1.0, -big_m[i]], -np.inf, 0.0, f"trade_{i}")
|
||||||
|
|
||||||
|
# 11. target expected P&L for min_cvar
|
||||||
|
pnl_idx = np.concatenate([self.i_n, cost_idx])
|
||||||
|
pnl_coef = np.concatenate([mean_gain, -cost_coef])
|
||||||
|
if sp_.objective == "min_cvar":
|
||||||
|
add_row(pnl_idx, pnl_coef, float(sp_.target_pnl), np.inf, "target_pnl")
|
||||||
|
|
||||||
|
# objective
|
||||||
|
c = np.zeros(self.n_vars)
|
||||||
|
if sp_.objective == "max_return":
|
||||||
|
np.add.at(c, pnl_idx, pnl_coef)
|
||||||
|
maximize = True
|
||||||
|
elif sp_.objective == "min_cvar":
|
||||||
|
np.add.at(c, cvar_idx, cvar_coef)
|
||||||
|
maximize = False
|
||||||
|
else: # utility
|
||||||
|
np.add.at(c, pnl_idx, pnl_coef)
|
||||||
|
np.add.at(c, cvar_idx, -sp_.risk_aversion * cvar_coef)
|
||||||
|
maximize = True
|
||||||
|
|
||||||
|
# variable bounds and types
|
||||||
|
var_lb = np.zeros(self.n_vars)
|
||||||
|
var_ub = np.full(self.n_vars, np.inf)
|
||||||
|
types = ["C"] * self.n_vars
|
||||||
|
names = [""] * self.n_vars
|
||||||
|
for i in range(n):
|
||||||
|
types[self.i_n[i]] = "I"
|
||||||
|
names[self.i_n[i]] = f"n[{sp_.symbols[i]}]"
|
||||||
|
var_ub[self.i_n[i]] = ub_shares[i]
|
||||||
|
if self.rebalancing:
|
||||||
|
for i in range(n):
|
||||||
|
types[self.i_b[i]] = types[self.i_s[i]] = "I"
|
||||||
|
names[self.i_b[i]] = f"buy[{sp_.symbols[i]}]"
|
||||||
|
names[self.i_s[i]] = f"sell[{sp_.symbols[i]}]"
|
||||||
|
var_ub[self.i_b[i]] = ub_shares[i]
|
||||||
|
var_ub[self.i_s[i]] = sp_.holdings[i]
|
||||||
|
max_gain, max_loss = self._loss_bounds(ub_shares)
|
||||||
|
if self.gaussian:
|
||||||
|
var_lb[self.i_cv], var_ub[self.i_cv] = -max_gain, max_loss
|
||||||
|
names[self.i_cv] = "cvar"
|
||||||
|
else:
|
||||||
|
var_lb[self.i_alpha], var_ub[self.i_alpha] = -max_gain, max_loss
|
||||||
|
names[self.i_alpha] = "alpha"
|
||||||
|
for s in range(S):
|
||||||
|
var_ub[self.i_u[s]] = max_loss + max_gain # u_s <= L_s - alpha over the box
|
||||||
|
names[self.i_u[s]] = f"u[{s}]"
|
||||||
|
if self.use_pos_bin:
|
||||||
|
for i in range(n):
|
||||||
|
types[self.i_y[i]] = "I"
|
||||||
|
var_ub[self.i_y[i]] = 1.0
|
||||||
|
names[self.i_y[i]] = f"y[{sp_.symbols[i]}]"
|
||||||
|
if self.use_trade_bin:
|
||||||
|
for i in range(n):
|
||||||
|
types[self.i_t[i]] = "I"
|
||||||
|
var_ub[self.i_t[i]] = 1.0
|
||||||
|
names[self.i_t[i]] = f"t[{sp_.symbols[i]}]"
|
||||||
|
|
||||||
|
A = sp.csr_matrix((vals, (rows, cols)), shape=(r, self.n_vars))
|
||||||
|
A.sum_duplicates()
|
||||||
|
A.sort_indices()
|
||||||
|
return MILPModel(c=c, A=A, row_lb=np.array(row_lb, dtype=float), row_ub=np.array(row_ub, dtype=float),
|
||||||
|
var_lb=var_lb, var_ub=var_ub, var_types=types, var_names=names, row_names=row_names,
|
||||||
|
maximize=maximize)
|
||||||
|
|
||||||
|
# -- decode -----------------------------------------------------------------------------------
|
||||||
|
def decode(self, x: np.ndarray) -> dict:
|
||||||
|
"""Turn a solution vector into shares, trades, cash, costs and the realised CVaR."""
|
||||||
|
sp_ = self.spec
|
||||||
|
p = sp_.prices
|
||||||
|
shares = np.rint(x[self.i_n]).astype(int)
|
||||||
|
if self.rebalancing:
|
||||||
|
buys = np.rint(x[self.i_b]).astype(int)
|
||||||
|
sells = np.rint(x[self.i_s]).astype(int)
|
||||||
|
else:
|
||||||
|
buys, sells = shares.copy(), np.zeros(self.n, dtype=int)
|
||||||
|
trade_value_buy = buys * p
|
||||||
|
trade_value_sell = sells * p
|
||||||
|
cost = (sp_.costs.proportional_buy * trade_value_buy + sp_.costs.proportional_sell * trade_value_sell)
|
||||||
|
if self.use_trade_bin:
|
||||||
|
cost = cost + sp_.costs.fixed_per_trade * ((buys + sells) > 0)
|
||||||
|
total_cost = float(cost.sum())
|
||||||
|
cash_end = float(sp_.cash + trade_value_sell.sum() - trade_value_buy.sum() - total_cost)
|
||||||
|
value = shares * p
|
||||||
|
expected_pnl = float(sp_.mean_returns @ value - total_cost) # the objective's expectation
|
||||||
|
out = {
|
||||||
|
"shares": shares, "buys": buys, "sells": sells, "trades": buys - sells, "value": value,
|
||||||
|
"cost": cost, "total_cost": total_cost, "cash_end": cash_end, "invested": float(value.sum()),
|
||||||
|
"expected_pnl": expected_pnl, "weights": value / sp_.wealth,
|
||||||
|
"turnover": float((trade_value_buy.sum() + trade_value_sell.sum()) / sp_.wealth),
|
||||||
|
}
|
||||||
|
if self.gaussian:
|
||||||
|
var_m, cvar_m, sigma = self.gaussian_risk(value, total_cost)
|
||||||
|
out.update({"var": var_m, "cvar": cvar_m, "sigma": sigma, "alpha": var_m,
|
||||||
|
"cvar_model": float(x[self.i_cv]), "pnl_scenarios": None})
|
||||||
|
else:
|
||||||
|
pnl = sp_.scenarios @ value - total_cost # money P&L per scenario
|
||||||
|
var_m, cvar_m = scenario_cvar(-pnl, sp_.confidence, sp_.probabilities)
|
||||||
|
w = sp_.weights
|
||||||
|
out.update({"var": var_m, "cvar": cvar_m, "alpha": float(x[self.i_alpha]), "pnl_scenarios": pnl,
|
||||||
|
"expected_pnl_scenarios": float((w @ pnl) / w.sum())})
|
||||||
|
return out
|
||||||
151
quantcore/optimization/rowgen.py
Normal file
151
quantcore/optimization/rowgen.py
Normal file
@ -0,0 +1,151 @@
|
|||||||
|
"""Tail-row generation for large scenario sets (Kuenzi-Bay & Mayer, 2006).
|
||||||
|
|
||||||
|
In the Rockafellar-Uryasev MILP only the scenarios in the tail have ``u_s > 0`` at the optimum;
|
||||||
|
every other scenario row is slack. So the problem is solved on a *working set* of scenarios
|
||||||
|
(initially the worst ``initial_multiple * (1 - beta) * S`` scenarios of a reference portfolio),
|
||||||
|
the losses of all ``S`` scenarios are evaluated for the solution, the scenarios whose loss exceeds
|
||||||
|
the solution's ``alpha`` are added, and the loop repeats. The restricted problem is a relaxation
|
||||||
|
(dropped rows, ``u_s = 0``), so its objective bounds the full one; when no omitted scenario is
|
||||||
|
violated the solution is optimal for the full scenario set (within the MIP gap).
|
||||||
|
|
||||||
|
Measured on the real universe (30 stocks, beta = 0.95): working sets of ~11 % of ``S``, two to
|
||||||
|
three iterations, 6-7x faster than the full MILP at 32 768-100 000 scenarios -- and only the
|
||||||
|
working-set rows are sent to the solver.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
from collections.abc import Callable
|
||||||
|
from dataclasses import dataclass, field, replace
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from quantcore.optimization.problem import MeanCVaRProblem, MILPModel, ProblemSpec
|
||||||
|
from quantcore.optimization.solvers import SolverResult
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
SolveFn = Callable[[MILPModel], SolverResult]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class RowGenerationInfo:
|
||||||
|
iterations: int = 0
|
||||||
|
working_set_sizes: list[int] = field(default_factory=list)
|
||||||
|
violated: list[int] = field(default_factory=list)
|
||||||
|
solve_times: list[float] = field(default_factory=list)
|
||||||
|
total_time: float = 0.0
|
||||||
|
converged: bool = False
|
||||||
|
|
||||||
|
def as_dict(self) -> dict:
|
||||||
|
return {"iterations": self.iterations, "working_set_sizes": self.working_set_sizes, "violated": self.violated,
|
||||||
|
"rows_final": self.working_set_sizes[-1] if self.working_set_sizes else 0,
|
||||||
|
"solve_times_s": [round(t, 3) for t in self.solve_times], "total_time_s": round(self.total_time, 3),
|
||||||
|
"converged": self.converged}
|
||||||
|
|
||||||
|
|
||||||
|
def reference_exposure(spec: ProblemSpec) -> np.ndarray:
|
||||||
|
"""Money exposure of the reference portfolio: the current holdings, or (new portfolio) an
|
||||||
|
equal-weight allocation within the per-stock cap."""
|
||||||
|
if spec.rebalancing:
|
||||||
|
return spec.holdings * spec.prices
|
||||||
|
n = len(spec.symbols)
|
||||||
|
w = min(spec.constraints.max_weight, spec.constraints.max_invested, 1.0 / n)
|
||||||
|
return np.full(n, w * spec.wealth)
|
||||||
|
|
||||||
|
|
||||||
|
def scenario_losses(spec: ProblemSpec, exposure: np.ndarray, total_cost: float = 0.0) -> np.ndarray:
|
||||||
|
"""Money loss of every scenario for a given exposure (``TC - sum_i R_si v_i``)."""
|
||||||
|
return total_cost - spec.scenarios @ np.asarray(exposure, dtype=float)
|
||||||
|
|
||||||
|
|
||||||
|
def restricted_spec(spec: ProblemSpec, work: np.ndarray) -> ProblemSpec:
|
||||||
|
"""The spec on a subset of scenarios: probabilities keep their *full-set* values (the omitted
|
||||||
|
scenarios contribute ``u_s = 0``), the objective keeps the full-set expected returns."""
|
||||||
|
return replace(spec, scenarios=spec.scenarios[work], probabilities=spec.weights[work],
|
||||||
|
expected_returns=spec.mean_returns)
|
||||||
|
|
||||||
|
|
||||||
|
def expand_solution(problem_r: MeanCVaRProblem, x_r: np.ndarray, problem_full: MeanCVaRProblem) -> np.ndarray:
|
||||||
|
"""Map a restricted solution to the full layout: ``u_s = max(L_s - alpha, 0)`` for every scenario."""
|
||||||
|
x = np.zeros(problem_full.n_vars)
|
||||||
|
x[problem_full.i_n] = x_r[problem_r.i_n]
|
||||||
|
if problem_full.rebalancing:
|
||||||
|
x[problem_full.i_b] = x_r[problem_r.i_b]
|
||||||
|
x[problem_full.i_s] = x_r[problem_r.i_s]
|
||||||
|
alpha = float(x_r[problem_r.i_alpha])
|
||||||
|
x[problem_full.i_alpha] = alpha
|
||||||
|
d = problem_r.decode(x_r)
|
||||||
|
losses = scenario_losses(problem_full.spec, d["value"], d["total_cost"])
|
||||||
|
x[problem_full.i_u] = np.maximum(losses - alpha, 0.0)
|
||||||
|
if problem_full.use_pos_bin:
|
||||||
|
x[problem_full.i_y] = x_r[problem_r.i_y]
|
||||||
|
if problem_full.use_trade_bin:
|
||||||
|
x[problem_full.i_t] = x_r[problem_r.i_t]
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
def solve_by_row_generation(spec: ProblemSpec, solve: SolveFn, initial_multiple: float = 2.0, max_iter: int = 30,
|
||||||
|
tol: float = 1e-7, deadline: float | None = None
|
||||||
|
) -> tuple[SolverResult, MeanCVaRProblem, RowGenerationInfo]:
|
||||||
|
"""Solve the scenario MILP of ``spec`` by tail-row generation.
|
||||||
|
|
||||||
|
``solve`` maps a :class:`MILPModel` to a :class:`SolverResult`. Returns the result in the
|
||||||
|
*full* problem's layout (so :meth:`MeanCVaRProblem.decode` of the returned problem works), the
|
||||||
|
full problem and the iteration log.
|
||||||
|
|
||||||
|
``deadline`` is a :func:`time.monotonic` timestamp: the *request's* time budget. Each pass is a
|
||||||
|
MILP of its own, so without it a limit of ``T`` seconds would allow ``T`` per pass instead of
|
||||||
|
``T`` in total. Once it passes, the current (feasible, not proven optimal) solution is returned.
|
||||||
|
"""
|
||||||
|
if spec.is_gaussian:
|
||||||
|
raise ValueError("row generation applies to scenario problems")
|
||||||
|
t0 = time.monotonic()
|
||||||
|
info = RowGenerationInfo()
|
||||||
|
S = spec.n_scenarios
|
||||||
|
full = MeanCVaRProblem(spec)
|
||||||
|
losses0 = scenario_losses(spec, reference_exposure(spec))
|
||||||
|
k0 = int(min(S, max(np.ceil(initial_multiple * (1.0 - spec.confidence) * S), 2)))
|
||||||
|
work = np.sort(np.argsort(losses0)[-k0:])
|
||||||
|
scale = max(abs(spec.cvar_limit or 0.0), spec.wealth * 1e-3, 1.0)
|
||||||
|
result: SolverResult | None = None
|
||||||
|
for it in range(1, max_iter + 1):
|
||||||
|
problem_r = MeanCVaRProblem(restricted_spec(spec, work))
|
||||||
|
t1 = time.monotonic()
|
||||||
|
result = solve(problem_r.build())
|
||||||
|
info.solve_times.append(time.monotonic() - t1)
|
||||||
|
info.working_set_sizes.append(len(work))
|
||||||
|
info.iterations = it
|
||||||
|
if not result.ok:
|
||||||
|
log.warning("row generation: solver returned %s at iteration %d", result.status, it)
|
||||||
|
info.total_time = time.monotonic() - t0
|
||||||
|
return result, full, info
|
||||||
|
d = problem_r.decode(result.x)
|
||||||
|
alpha = float(result.x[problem_r.i_alpha])
|
||||||
|
losses = scenario_losses(spec, d["value"], d["total_cost"])
|
||||||
|
violated = np.setdiff1d(np.where(losses > alpha + tol * scale)[0], work)
|
||||||
|
info.violated.append(len(violated))
|
||||||
|
log.info("row generation: iteration %d, working set %d of %d, %d violated", it, len(work), S, len(violated))
|
||||||
|
if len(violated) > 0 and deadline is not None and time.monotonic() >= deadline:
|
||||||
|
log.warning("row generation: time budget spent after %d iterations (%d rows still violated)",
|
||||||
|
it, len(violated))
|
||||||
|
x_full = expand_solution(problem_r, result.x, full)
|
||||||
|
info.total_time = time.monotonic() - t0
|
||||||
|
return replace(result, x=x_full, status="FeasibleFound",
|
||||||
|
message=(result.message + f" (row generation: time limit after {it} iterations, "
|
||||||
|
f"{len(work)} of {S} rows)").strip()), full, info
|
||||||
|
if len(violated) == 0:
|
||||||
|
info.converged = True
|
||||||
|
x_full = expand_solution(problem_r, result.x, full)
|
||||||
|
result = replace(result, x=x_full, message=(result.message + f" (row generation: {it} iterations, "
|
||||||
|
f"{len(work)} of {S} rows)").strip())
|
||||||
|
info.total_time = time.monotonic() - t0
|
||||||
|
return result, full, info
|
||||||
|
work = np.union1d(work, violated)
|
||||||
|
log.warning("row generation did not converge in %d iterations", max_iter)
|
||||||
|
x_full = expand_solution(problem_r, result.x, full)
|
||||||
|
info.total_time = time.monotonic() - t0
|
||||||
|
return replace(result, x=x_full, status="FeasibleFound",
|
||||||
|
message=(result.message + " (row generation: iteration limit)").strip()), full, info
|
||||||
763
quantcore/optimization/service.py
Normal file
763
quantcore/optimization/service.py
Normal file
@ -0,0 +1,763 @@
|
|||||||
|
"""``OptimizationService`` -- from a request to an integer-share portfolio proposal.
|
||||||
|
|
||||||
|
Pipeline (deterministic given the seed; the result carries the scenario hash and a
|
||||||
|
``reproducible`` flag):
|
||||||
|
|
||||||
|
1. candidate stocks (a list or the largest ``max_assets`` of the universe by market cap);
|
||||||
|
2. expected returns: Black-Litterman on 10 years of weekly CHF returns with market-cap
|
||||||
|
weights, the broad-market average as the equilibrium return, and Idzorek-confidence views
|
||||||
|
(or plain historical means / zero drift) -- *arithmetic* annual expected returns;
|
||||||
|
3. horizon scenarios from the chosen stock-price model (GBM or empirical innovations) with
|
||||||
|
the chosen covariance estimator: scrambled Sobol' draws, moment matching (exact mean and
|
||||||
|
covariance), optionally an importance-sampling tilt along the loss direction of a first-pass
|
||||||
|
solution (``importance_sampling``), or -- ``scenario_model='gaussian_exact'`` -- no scenarios
|
||||||
|
at all but the exact Gaussian (mean-variance-equivalent) problem by outer approximation;
|
||||||
|
4. the mean-CVaR MILP (:mod:`quantcore.optimization.problem`) in "new" or "rebalancing" mode,
|
||||||
|
for large scenario sets by tail-row generation (:mod:`quantcore.optimization.rowgen`);
|
||||||
|
5. solved on cuOpt (HiGHS fallback), decoded into shares, trades, cash, costs, risk figures.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
from collections import OrderedDict
|
||||||
|
from dataclasses import asdict, dataclass, field
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
from scipy.stats import norm
|
||||||
|
|
||||||
|
from quantcore.config import Settings
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.measurement.returns import horizon_periods, periods_per_year
|
||||||
|
from quantcore.measurement.risk import scenario_cvar
|
||||||
|
from quantcore.models.black_litterman import (
|
||||||
|
BlackLittermanModel,
|
||||||
|
BlackLittermanResult,
|
||||||
|
View,
|
||||||
|
view_from_sentiment,
|
||||||
|
)
|
||||||
|
from quantcore.models.covariance import CovarianceModel
|
||||||
|
from quantcore.models.sampling import next_power_of_two, scenario_hash
|
||||||
|
from quantcore.models.simulation import ScenarioSet, TiltSpec, make_simulator
|
||||||
|
from quantcore.optimization.gaussian_exact import solve_gaussian_exact
|
||||||
|
from quantcore.optimization.problem import (
|
||||||
|
MeanCVaRProblem,
|
||||||
|
OptimizationConstraints,
|
||||||
|
ProblemSpec,
|
||||||
|
TransactionCosts,
|
||||||
|
cvar_multiplier,
|
||||||
|
)
|
||||||
|
from quantcore.optimization.rowgen import solve_by_row_generation
|
||||||
|
from quantcore.optimization.solvers import HighsSolver, SolverResult, solve_with_fallback
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
SCENARIO_MODELS = ("gbm", "empirical", "gaussian_exact")
|
||||||
|
IMPORTANCE_SAMPLING_MODES = ("auto", "market", "adaptive", "off")
|
||||||
|
ROW_GENERATION_MIN_SCENARIOS = 8192 # 'auto': row generation from this size on
|
||||||
|
IMPORTANCE_SAMPLING_MIN_TAIL = 128 # 'auto': tilt when fewer than this many scenarios would be in the tail
|
||||||
|
STAGE1_SCENARIOS = 2048 # 'adaptive': first-pass solve that provides the tilt direction
|
||||||
|
TILT_ALIGNMENT_WARN = 0.3 # warn below this correlation between the solution's loss and the tilt's
|
||||||
|
SCENARIO_CACHE_SIZE = 8 # scenario sets kept per service instance
|
||||||
|
# The sample-average approximation searches an N-dimensional space, so the scenario count has to
|
||||||
|
# grow with the number of candidates. Measured on the real universe (GBM model against the exact
|
||||||
|
# optimum): the *reported figures* are stable at any size (the standard deviation of the expected
|
||||||
|
# P&L is below 0.1 % from 2 048 scenarios on), but the *composition* keeps moving between
|
||||||
|
# near-equivalent portfolios -- at 2 048 scenarios two seeds differ by 3.5 % of wealth for 30
|
||||||
|
# candidates and 12.5 % for 100. About 64 scenarios per candidate brings that near 5 %, which is
|
||||||
|
# too expensive as a default (N = 300 would mean 32 768 scenarios and minutes per solve), so the
|
||||||
|
# default is 16 per candidate and the result says when the ratio is low.
|
||||||
|
SCENARIOS_PER_ASSET = 16
|
||||||
|
MIN_DEFAULT_SCENARIOS = 4096
|
||||||
|
MAX_DEFAULT_SCENARIOS = 16384 # beyond this prefer 'gaussian_exact' (no scenarios) or set it explicitly
|
||||||
|
SCENARIOS_PER_ASSET_WARN = 8 # warn below this ratio
|
||||||
|
|
||||||
|
|
||||||
|
def _view_fingerprint(views) -> list:
|
||||||
|
"""Views as plain sorted data (for the adaptive-tilt part of the scenario key)."""
|
||||||
|
out = []
|
||||||
|
for v in views:
|
||||||
|
d = v if isinstance(v, dict) else {"assets": dict(v.assets), "q": v.q, "confidence": v.confidence}
|
||||||
|
out.append({k: d[k] for k in sorted(d) if k not in ("label", "source")})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class OptimizationRequest:
|
||||||
|
mode: str = "new" # 'new' (all from cash) | 'rebalance' (existing holdings + cash)
|
||||||
|
cash: float = 1_000_000.0
|
||||||
|
holdings: dict[str, int] = field(default_factory=dict)
|
||||||
|
symbols: list[str] | None = None # candidate stocks; default: largest `max_assets` by market cap
|
||||||
|
max_assets: int = 100
|
||||||
|
horizon: str | int = "week"
|
||||||
|
freq: str = "daily"
|
||||||
|
confidence: float = 0.95
|
||||||
|
cvar_limit: float | None = None # money over the horizon
|
||||||
|
cvar_limit_fraction: float | None = None # alternative: fraction of total wealth
|
||||||
|
objective: str = "max_return" # 'max_return' | 'min_cvar' | 'utility'
|
||||||
|
risk_aversion: float = 1.0
|
||||||
|
target_return: float | None = None # fraction of wealth over the horizon (objective 'min_cvar')
|
||||||
|
scenario_model: str = "empirical" # 'gbm' | 'empirical' | 'gaussian_exact'
|
||||||
|
n_scenarios: int | None = None # None: 16 per candidate, within [4096, 16384] (see scenario_count)
|
||||||
|
seed: int = 42
|
||||||
|
sampling: str = "sobol" # 'sobol' | 'mc' | 'antithetic' | 'lhs' (gbm); 'sobol' | 'mc' | 'balanced' (empirical)
|
||||||
|
moment_match: bool = True # exact scenario mean and covariance
|
||||||
|
# 'market': tilt along the loss direction of the cap-weighted candidate portfolio, in blocks at
|
||||||
|
# z_0.95 and z_0.99 -- independent of the risk limit, the constraints and the confidence, so a
|
||||||
|
# family of runs shares one scenario set; 'adaptive': tilt towards the tail of a first-pass
|
||||||
|
# solution (slightly sharper, but the set then depends on those parameters); 'auto': 'market'
|
||||||
|
# when the tail would otherwise be thin; 'off': no tilt.
|
||||||
|
importance_sampling: str = "auto"
|
||||||
|
is_mixture: float = 0.25 # share of scenarios per tilted block ('adaptive': of the single block)
|
||||||
|
drift_convention: str = "arithmetic" # 'arithmetic' (Ito-corrected log-drift) | 'log'
|
||||||
|
# 'model': the CVaR is measured on the scenarios as they are, drift included -- over a long
|
||||||
|
# horizon a large expected return then shifts the whole tail into profit and the risk limit
|
||||||
|
# stops binding. 'zero': the risk rows use the centred (driftless) scenarios while the
|
||||||
|
# objective keeps the expected return, which is the usual market-risk convention.
|
||||||
|
risk_drift: str = "model" # 'model' | 'zero'
|
||||||
|
row_generation: str = "auto" # 'auto' | 'on' | 'off': solve on the tail rows only
|
||||||
|
covariance: dict = field(default_factory=lambda: {"method": "ewm", "lam": 0.99, "shrinkage": "auto"})
|
||||||
|
expected_returns: str = "black_litterman" # 'black_litterman' | 'historical' | 'zero'
|
||||||
|
views: list[dict] = field(default_factory=list)
|
||||||
|
bl_tau: float = 0.05
|
||||||
|
bl_years: int = 10
|
||||||
|
bl_omega_method: str = "idzorek" # 'idzorek' | 'he_litterman'
|
||||||
|
constraints: OptimizationConstraints = field(default_factory=OptimizationConstraints)
|
||||||
|
costs: TransactionCosts = field(default_factory=TransactionCosts)
|
||||||
|
solver: str = "auto" # 'auto' | 'cuopt' | 'highs'
|
||||||
|
time_limit: float | None = None
|
||||||
|
innovation_history: int | None = 2500 # observations used for the empirical innovations
|
||||||
|
innovation_standardize: str | None = "diagonal" # 'diagonal' | 'full' | None (see simulation module)
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
if self.mode not in ("new", "rebalance"):
|
||||||
|
raise ValueError("mode must be 'new' or 'rebalance'")
|
||||||
|
if self.mode == "new" and self.holdings:
|
||||||
|
raise ValueError("mode 'new' starts from cash only - pass no holdings")
|
||||||
|
if self.scenario_model not in SCENARIO_MODELS:
|
||||||
|
raise ValueError(f"scenario_model must be one of {SCENARIO_MODELS}")
|
||||||
|
if self.importance_sampling not in IMPORTANCE_SAMPLING_MODES:
|
||||||
|
raise ValueError(f"importance_sampling must be one of {IMPORTANCE_SAMPLING_MODES}")
|
||||||
|
if self.row_generation not in ("auto", "on", "off"):
|
||||||
|
raise ValueError("row_generation must be 'auto', 'on' or 'off'")
|
||||||
|
if self.risk_drift not in ("model", "zero"):
|
||||||
|
raise ValueError("risk_drift must be 'model' or 'zero'")
|
||||||
|
if isinstance(self.constraints, dict):
|
||||||
|
self.constraints = OptimizationConstraints(**self.constraints)
|
||||||
|
if isinstance(self.costs, dict):
|
||||||
|
self.costs = TransactionCosts(**self.costs)
|
||||||
|
if self.cvar_limit is None and self.cvar_limit_fraction is None and self.objective == "max_return":
|
||||||
|
self.cvar_limit_fraction = 0.05
|
||||||
|
log.info("no CVaR limit given - using 5%% of wealth over the horizon")
|
||||||
|
|
||||||
|
@property
|
||||||
|
def horizon_periods(self) -> int:
|
||||||
|
return horizon_periods(self.horizon, self.freq)
|
||||||
|
|
||||||
|
def scenario_count(self, n_assets: int) -> int:
|
||||||
|
"""Scenarios to draw for ``n_assets`` candidates (the request's value, or the default rule)."""
|
||||||
|
if self.n_scenarios is not None:
|
||||||
|
return int(self.n_scenarios)
|
||||||
|
target = SCENARIOS_PER_ASSET * max(int(n_assets), 1)
|
||||||
|
return int(min(max(next_power_of_two(target), MIN_DEFAULT_SCENARIOS), MAX_DEFAULT_SCENARIOS))
|
||||||
|
|
||||||
|
def uses_row_generation(self, n_scenarios: int) -> bool:
|
||||||
|
if self.row_generation == "on":
|
||||||
|
return True
|
||||||
|
return self.row_generation == "auto" and n_scenarios >= ROW_GENERATION_MIN_SCENARIOS
|
||||||
|
|
||||||
|
def tilt_mode(self, n_scenarios: int) -> str:
|
||||||
|
"""``'market'``, ``'adaptive'`` or ``'off'`` for this request.
|
||||||
|
|
||||||
|
``'auto'`` is ``'market'``: the tilt blocks cover both usual confidence levels, it costs
|
||||||
|
nothing in accuracy at 95 % and roughly halves the tail noise at 99 % -- and, unlike a
|
||||||
|
rule that switches on the tail count, it keeps the scenario set independent of
|
||||||
|
``confidence``, so runs at different risk levels stay comparable.
|
||||||
|
"""
|
||||||
|
if self.importance_sampling in ("market", "adaptive", "off"):
|
||||||
|
return self.importance_sampling
|
||||||
|
return "market"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class OptimizationResult:
|
||||||
|
status: str
|
||||||
|
solver: str
|
||||||
|
mode: str
|
||||||
|
as_of: str
|
||||||
|
wealth: float
|
||||||
|
cash_start: float
|
||||||
|
cash_end: float
|
||||||
|
invested: float
|
||||||
|
total_cost: float
|
||||||
|
turnover: float
|
||||||
|
expected_pnl: float
|
||||||
|
expected_return: float # fraction of wealth over the horizon
|
||||||
|
var: float # money
|
||||||
|
cvar: float # money
|
||||||
|
cvar_limit: float | None
|
||||||
|
confidence: float
|
||||||
|
horizon: str | int
|
||||||
|
horizon_periods: int
|
||||||
|
n_positions: int
|
||||||
|
n_candidates: int
|
||||||
|
positions: list[dict]
|
||||||
|
sector_exposure: dict
|
||||||
|
model: dict
|
||||||
|
solver_info: dict
|
||||||
|
timings: dict
|
||||||
|
reproducible: bool = False # optimality proven -> same inputs give the same figures
|
||||||
|
reproducibility: str = "none" # 'exact' (HiGHS: identical shares) | 'within_gap' (cuOpt: same value, ties may differ)
|
||||||
|
scenario_hash: str | None = None
|
||||||
|
scenario_key: dict | None = None # the inputs the scenario set depends on (see OptimizationService.scenario_key)
|
||||||
|
before: dict | None = None # rebalancing: the starting portfolio measured on the same scenarios
|
||||||
|
warnings: list[str] = field(default_factory=list)
|
||||||
|
|
||||||
|
def as_dict(self) -> dict:
|
||||||
|
return asdict(self)
|
||||||
|
|
||||||
|
def positions_frame(self) -> pd.DataFrame:
|
||||||
|
return pd.DataFrame(self.positions).set_index("symbol") if self.positions else pd.DataFrame()
|
||||||
|
|
||||||
|
|
||||||
|
class OptimizationService:
|
||||||
|
def __init__(self, settings: Settings, market_data: MarketDataService | None = None):
|
||||||
|
self.settings = settings
|
||||||
|
self.md = market_data or MarketDataService(settings)
|
||||||
|
self._scenario_cache: OrderedDict[str, ScenarioSet] = OrderedDict()
|
||||||
|
|
||||||
|
# -- building blocks --------------------------------------------------------------------------
|
||||||
|
def candidate_symbols(self, req: OptimizationRequest) -> list[str]:
|
||||||
|
if req.symbols:
|
||||||
|
symbols = list(dict.fromkeys(req.symbols))
|
||||||
|
else:
|
||||||
|
caps = self.md.market_caps().sort_values(ascending=False)
|
||||||
|
symbols = caps.index[: req.max_assets].tolist()
|
||||||
|
for s in req.holdings: # existing holdings must be in the candidate set (they can be sold)
|
||||||
|
if s not in symbols:
|
||||||
|
symbols.append(s)
|
||||||
|
for v in req.views: # so must the assets a view refers to
|
||||||
|
assets = [v["symbol"]] if isinstance(v, dict) and "symbol" in v else (
|
||||||
|
list(v["assets"]) if isinstance(v, dict) else list(v.assets))
|
||||||
|
for s in assets:
|
||||||
|
if s not in symbols:
|
||||||
|
symbols.append(s)
|
||||||
|
unknown = [s for s in symbols if s not in self.md.universe.index]
|
||||||
|
if unknown:
|
||||||
|
raise KeyError(f"symbols not in the universe: {unknown}")
|
||||||
|
return symbols
|
||||||
|
|
||||||
|
def expected_returns(self, req: OptimizationRequest, symbols: list[str]) -> tuple[pd.Series, dict, BlackLittermanResult | None]:
|
||||||
|
"""Annual (arithmetic) expected returns per symbol and a description of the model."""
|
||||||
|
last = self.md.last_date()
|
||||||
|
start = (last - pd.DateOffset(years=req.bl_years)).date().isoformat()
|
||||||
|
if req.expected_returns == "zero":
|
||||||
|
return pd.Series(0.0, index=symbols), {"type": "zero"}, None
|
||||||
|
weekly = self.md.returns(symbols, start=start, currency="base", freq="weekly").dropna()
|
||||||
|
if req.expected_returns == "historical":
|
||||||
|
mu = weekly.mean() * 52 + 0.5 * weekly.var() * 52 # arithmetic: log mean plus sigma^2/2
|
||||||
|
return mu, {"type": "historical", "years": req.bl_years, "n_weeks": len(weekly)}, None
|
||||||
|
if req.expected_returns != "black_litterman":
|
||||||
|
raise ValueError("expected_returns must be 'black_litterman', 'historical' or 'zero'")
|
||||||
|
cov_w = weekly.cov()
|
||||||
|
market_w = self.md.market_weights(symbols)
|
||||||
|
index_r = self.md.cap_weighted_index(start=start, freq="weekly")
|
||||||
|
market_return = float(index_r.mean() + 0.5 * index_r.var()) # arithmetic weekly market return
|
||||||
|
bl = BlackLittermanModel(cov_w, market_w, market_return, risk_free=self.settings.risk_free_rate / 52,
|
||||||
|
tau=req.bl_tau, periods_per_year=52)
|
||||||
|
views = self._views(req, bl)
|
||||||
|
result = bl.posterior(views, omega_method=req.bl_omega_method)
|
||||||
|
desc = {
|
||||||
|
"type": "black_litterman", "years": req.bl_years, "n_weeks": len(weekly),
|
||||||
|
"market_return_ann": result.market_return_ann, "market_volatility_ann": result.market_volatility_ann,
|
||||||
|
"delta": result.delta, "tau": result.tau, "views": result.views, "convention": "arithmetic",
|
||||||
|
"omega_method": req.bl_omega_method,
|
||||||
|
}
|
||||||
|
return result.posterior_ann, desc, result
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _views(req: OptimizationRequest, bl: BlackLittermanModel) -> list[View]:
|
||||||
|
views = []
|
||||||
|
for v in req.views:
|
||||||
|
if isinstance(v, View):
|
||||||
|
views.append(v)
|
||||||
|
elif "sentiment" in v:
|
||||||
|
views.append(view_from_sentiment(bl, v["symbol"], float(v["sentiment"]), float(v.get("confidence", 0.5)),
|
||||||
|
float(v.get("strength", 0.25)), v.get("label", ""), v.get("source", "")))
|
||||||
|
else:
|
||||||
|
views.append(View(assets=v["assets"], q=float(v["q"]), confidence=float(v.get("confidence", 0.5)),
|
||||||
|
label=v.get("label", ""), source=v.get("source", "")))
|
||||||
|
return views
|
||||||
|
|
||||||
|
def _returns_and_cov_model(self, req: OptimizationRequest, symbols: list[str]) -> tuple[pd.DataFrame, CovarianceModel]:
|
||||||
|
r = self.md.returns(symbols, currency="base", freq=req.freq)
|
||||||
|
return r.dropna(), CovarianceModel.from_dict(req.covariance) # complete rows only for the estimators
|
||||||
|
|
||||||
|
def market_exposure(self, symbols: list[str]) -> np.ndarray:
|
||||||
|
"""Money exposure of the cap-weighted portfolio of the candidates (scale is irrelevant: only
|
||||||
|
its *direction* is used). This is the reference portfolio of the ``'market'`` tilt -- it
|
||||||
|
depends on the universe and the market caps, never on the request's risk parameters."""
|
||||||
|
w = self.md.market_weights(symbols).reindex(symbols).fillna(0.0).values
|
||||||
|
if w.sum() <= 0:
|
||||||
|
w = np.ones(len(symbols))
|
||||||
|
return w / w.sum()
|
||||||
|
|
||||||
|
def scenario_key(self, req: OptimizationRequest, symbols: list[str], tilt_mode: str | None = None,
|
||||||
|
n_scenarios: int | None = None) -> dict:
|
||||||
|
"""Exactly the inputs the scenario *shocks* depend on.
|
||||||
|
|
||||||
|
Two requests with the same key share their scenario set: the risk limit, the confidence
|
||||||
|
level, the constraints, the transaction costs, the objective, the cash and the holdings do
|
||||||
|
**not** appear here, so a whole family of runs is compared on identical scenarios. Neither
|
||||||
|
does the expected-return model: the drift is applied afterwards
|
||||||
|
(:meth:`ScenarioSet.with_drift`), so changing views shifts the scenarios without touching
|
||||||
|
their random part. The one exception is ``importance_sampling="adaptive"``, whose tilt
|
||||||
|
direction comes from a first-pass solution; the key then records the parameters of that
|
||||||
|
solve and runs with different risk parameters no longer share a set.
|
||||||
|
"""
|
||||||
|
S = int(n_scenarios or req.scenario_count(len(symbols)))
|
||||||
|
mode = tilt_mode if tilt_mode is not None else req.tilt_mode(S)
|
||||||
|
key = {
|
||||||
|
"symbols": list(symbols), "freq": req.freq, "horizon_periods": req.horizon_periods,
|
||||||
|
"model": req.scenario_model, "n_scenarios": S, "seed": req.seed,
|
||||||
|
"sampling": req.sampling, "moment_match": bool(req.moment_match),
|
||||||
|
"covariance": CovarianceModel.from_dict(req.covariance).describe(),
|
||||||
|
"importance_sampling": mode, "as_of": str(self.md.last_date().date()),
|
||||||
|
}
|
||||||
|
if mode != "off":
|
||||||
|
key["is_mixture"] = req.is_mixture
|
||||||
|
if mode == "adaptive": # the tilt depends on the optimisation parameters of the first pass
|
||||||
|
key["adaptive_tilt"] = {"confidence": req.confidence, "cvar_limit": req.cvar_limit,
|
||||||
|
"cvar_limit_fraction": req.cvar_limit_fraction, "objective": req.objective,
|
||||||
|
"cash": req.cash, "holdings": dict(sorted(req.holdings.items())),
|
||||||
|
"constraints": asdict(req.constraints), "costs": asdict(req.costs),
|
||||||
|
"expected_returns": req.expected_returns, "views": _view_fingerprint(req.views)}
|
||||||
|
if req.scenario_model in ("empirical", "innovations", "fhs"):
|
||||||
|
key["innovations"] = {"history": req.innovation_history, "standardize": req.innovation_standardize}
|
||||||
|
key["id"] = scenario_hash(np.frombuffer(json.dumps(key, sort_keys=True, default=str).encode(), dtype=np.uint8))
|
||||||
|
return key
|
||||||
|
|
||||||
|
def _generate_scenarios(self, req: OptimizationRequest, symbols: list[str], tilt: TiltSpec | None,
|
||||||
|
n_scenarios: int) -> ScenarioSet:
|
||||||
|
"""A drift-free scenario set (the expensive part); the drift is applied by the caller."""
|
||||||
|
r, cov_model = self._returns_and_cov_model(req, symbols)
|
||||||
|
kwargs = {}
|
||||||
|
if req.scenario_model in ("empirical", "innovations", "fhs"):
|
||||||
|
kwargs = {"max_history": req.innovation_history, "root": "symmetric" if len(symbols) <= 250 else "cholesky",
|
||||||
|
"standardize": req.innovation_standardize}
|
||||||
|
sim = make_simulator(req.scenario_model, cov_model, **kwargs)
|
||||||
|
scen = sim.simulate(r, n_scenarios=n_scenarios, horizon_periods=req.horizon_periods, mu=None, seed=req.seed,
|
||||||
|
freq=req.freq, sampling=req.sampling, moment_match=req.moment_match, tilt=tilt,
|
||||||
|
drift_convention=req.drift_convention)
|
||||||
|
scen.covariance_model["conditioning"] = cov_model.conditioning(r)
|
||||||
|
return scen
|
||||||
|
|
||||||
|
def scenarios(self, req: OptimizationRequest, symbols: list[str], mu_ann: pd.Series, tilt: TiltSpec | None = None,
|
||||||
|
n_scenarios: int | None = None, tilt_mode: str | None = None) -> ScenarioSet:
|
||||||
|
"""The scenario set for this request, with the expected returns applied as the drift.
|
||||||
|
|
||||||
|
The shocks are cached per :meth:`scenario_key`, so a series of runs that varies risk limits,
|
||||||
|
constraints, costs or views reuses one set (and pays the innovation filter only once).
|
||||||
|
"""
|
||||||
|
if req.scenario_model == "gaussian_exact":
|
||||||
|
raise ValueError("scenario_model 'gaussian_exact' does not generate scenarios")
|
||||||
|
S = int(n_scenarios or req.scenario_count(len(symbols)))
|
||||||
|
mode = tilt_mode if tilt_mode is not None else ("adaptive" if tilt is not None else req.tilt_mode(S))
|
||||||
|
if mode == "market" and tilt is None:
|
||||||
|
tilt = TiltSpec(exposure=self.market_exposure(symbols), mixture=(req.is_mixture,) * 2, label="market")
|
||||||
|
key = self.scenario_key(req, symbols, mode, S)
|
||||||
|
base = self._scenario_cache.get(key["id"])
|
||||||
|
hit = base is not None
|
||||||
|
if hit:
|
||||||
|
log.info("scenario cache hit (%s, %d scenarios)", key["id"], base.n_scenarios)
|
||||||
|
else:
|
||||||
|
base = self._generate_scenarios(req, symbols, tilt, S)
|
||||||
|
self._cache_scenarios(key["id"], base)
|
||||||
|
scen = base.with_drift(mu_ann / periods_per_year(req.freq), req.drift_convention)
|
||||||
|
scen.extra["key"] = key
|
||||||
|
scen.extra["cached"] = hit
|
||||||
|
return scen
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _sampling_warnings(n_assets: int, n_scenarios: int, cov_desc: dict) -> list[str]:
|
||||||
|
"""Scenario count and covariance conditioning against the size of the candidate set."""
|
||||||
|
out = []
|
||||||
|
if n_scenarios < SCENARIOS_PER_ASSET_WARN * n_assets:
|
||||||
|
out.append(
|
||||||
|
f"{n_scenarios} scenarios for {n_assets} candidates ({n_scenarios / n_assets:.0f} per asset): "
|
||||||
|
f"the expected P&L and CVaR are still reliable, but the composition will move between "
|
||||||
|
f"near-equivalent portfolios when the seed changes - raise n_scenarios (about 64 per "
|
||||||
|
f"candidate), reduce max_assets, or use scenario_model='gaussian_exact' for an exact answer")
|
||||||
|
cond = cov_desc.get("conditioning") or {}
|
||||||
|
if cond.get("rank_deficient"):
|
||||||
|
out.append(
|
||||||
|
f"the covariance estimate is singular ({cond['effective_observations']} effective "
|
||||||
|
f"observations for {cond['n_assets']} assets, shrinkage {cond['shrinkage_applied']}): some "
|
||||||
|
f"portfolios appear riskless to the optimiser - use shrinkage='auto' or a longer window")
|
||||||
|
elif cond.get("under_determined") and cond.get("min_eigenvalue_ratio", 1.0) < 1e-3:
|
||||||
|
out.append(
|
||||||
|
f"the covariance estimate is ill-conditioned (smallest eigenvalue "
|
||||||
|
f"{cond['min_eigenvalue_ratio']:.1e} of the average variance, {cond['effective_observations']} "
|
||||||
|
f"effective observations for {cond['n_assets']} assets) - consider shrinkage='auto'")
|
||||||
|
return out
|
||||||
|
|
||||||
|
def _cache_scenarios(self, key_id: str, scen: ScenarioSet) -> None:
|
||||||
|
self._scenario_cache[key_id] = scen
|
||||||
|
self._scenario_cache.move_to_end(key_id)
|
||||||
|
while len(self._scenario_cache) > SCENARIO_CACHE_SIZE:
|
||||||
|
self._scenario_cache.popitem(last=False)
|
||||||
|
|
||||||
|
def clear_scenario_cache(self) -> None:
|
||||||
|
self._scenario_cache.clear()
|
||||||
|
|
||||||
|
def _spec_common(self, req: OptimizationRequest, symbols: list[str]) -> dict:
|
||||||
|
prices = self.md.latest_prices(symbols, currency="base").reindex(symbols)
|
||||||
|
if prices.isna().any():
|
||||||
|
raise ValueError(f"no current price for {prices.index[prices.isna()].tolist()}")
|
||||||
|
holdings = np.array([float(req.holdings.get(s, 0)) for s in symbols])
|
||||||
|
wealth = req.cash + float(holdings @ prices.values)
|
||||||
|
cvar_limit = req.cvar_limit
|
||||||
|
if cvar_limit is None and req.cvar_limit_fraction is not None:
|
||||||
|
cvar_limit = req.cvar_limit_fraction * wealth
|
||||||
|
target_pnl = None if req.target_return is None else req.target_return * wealth
|
||||||
|
sectors = self.md.sectors(symbols).fillna("Unknown").tolist()
|
||||||
|
return dict(symbols=symbols, prices=prices.values, sectors=sectors, cash=req.cash, holdings=holdings,
|
||||||
|
confidence=req.confidence, cvar_limit=cvar_limit, objective=req.objective,
|
||||||
|
risk_aversion=req.risk_aversion, target_pnl=target_pnl, constraints=req.constraints, costs=req.costs)
|
||||||
|
|
||||||
|
def build_spec(self, req: OptimizationRequest, symbols: list[str], scen: ScenarioSet) -> ProblemSpec:
|
||||||
|
# risk_drift='zero': the CVaR rows see the shocks without the drift; the objective still
|
||||||
|
# uses the model's expected returns, so the forecast can never relax the risk limit
|
||||||
|
scenarios = np.expm1(scen.shocks) if req.risk_drift == "zero" else scen.simple_returns()
|
||||||
|
return ProblemSpec(scenarios=scenarios, probabilities=scen.probabilities,
|
||||||
|
expected_returns=scen.expected_simple, **self._spec_common(req, symbols))
|
||||||
|
|
||||||
|
def build_gaussian_spec(self, req: OptimizationRequest, symbols: list[str], mu_ann: pd.Series) -> tuple[ProblemSpec, dict]:
|
||||||
|
"""Gaussian mode: horizon simple returns ``N(mu * h / ppy, h * Sigma_T)``."""
|
||||||
|
ppy = periods_per_year(req.freq)
|
||||||
|
h = req.horizon_periods
|
||||||
|
r, cov_model = self._returns_and_cov_model(req, symbols)
|
||||||
|
cov = cov_model.estimate(r).values * h
|
||||||
|
mean = mu_ann.reindex(symbols).fillna(0.0).values / ppy * h
|
||||||
|
cov_desc = cov_model.describe()
|
||||||
|
cov_desc["conditioning"] = cov_model.conditioning(r)
|
||||||
|
desc = {"model": "gaussian_exact", "n_scenarios": 0, "n_assets": len(symbols), "horizon_periods": h,
|
||||||
|
"freq": req.freq, "as_of": str(r.index[-1].date()), "covariance_model": cov_desc,
|
||||||
|
"shrinkage_applied": cov_model.intensity(r), "drift_convention": "arithmetic"}
|
||||||
|
risk_mean = np.zeros_like(mean) if req.risk_drift == "zero" else mean
|
||||||
|
return ProblemSpec(scenarios=None, gaussian_mean=risk_mean, gaussian_cov=cov, expected_returns=mean,
|
||||||
|
**self._spec_common(req, symbols)), desc
|
||||||
|
|
||||||
|
def _solve_fn(self, req: OptimizationRequest, prefer_local: bool = False, deadline: float | None = None):
|
||||||
|
"""A solve function that spends at most the request's remaining time budget.
|
||||||
|
|
||||||
|
``time_limit`` is the budget of the *whole* request. Row generation and the Gaussian outer
|
||||||
|
approximation call the solver repeatedly, so each call gets what is left of it (at least a
|
||||||
|
second, since a solver cannot be given zero).
|
||||||
|
"""
|
||||||
|
def limit() -> float | None:
|
||||||
|
if deadline is None:
|
||||||
|
return req.time_limit
|
||||||
|
return max(1.0, deadline - time.monotonic())
|
||||||
|
|
||||||
|
# `prefer_local` covers the models cuOpt cannot be trusted with; see _solve_scenarios for
|
||||||
|
# the CVaR-objective case and `optimize` for the Gaussian cut MILPs. An explicit
|
||||||
|
# solver='cuopt' is still honoured -- the caller asked for it and gets cuOpt's own answer.
|
||||||
|
if prefer_local and req.solver in ("auto", "highs"):
|
||||||
|
return lambda model: HighsSolver(self.settings).solve(model, limit())
|
||||||
|
return lambda model: solve_with_fallback(model, self.settings, solver=req.solver, time_limit=limit())
|
||||||
|
|
||||||
|
def _deadline(self, req: OptimizationRequest) -> float | None:
|
||||||
|
return None if req.time_limit is None else time.monotonic() + float(req.time_limit)
|
||||||
|
|
||||||
|
def _solve_scenarios(self, req: OptimizationRequest, spec: ProblemSpec) -> tuple[SolverResult, MeanCVaRProblem, dict]:
|
||||||
|
deadline = self._deadline(req)
|
||||||
|
info: dict = {}
|
||||||
|
# cuOpt 26.08 aborts when the CVaR sits in the *objective* and the scenarios carry
|
||||||
|
# probabilities: the objective coefficients are then p_s/(1-beta), which importance sampling
|
||||||
|
# spreads over about six orders of magnitude (measured 1.1e-06 .. 1.0, ratio 8.8e5), and the
|
||||||
|
# LP relaxation aborts within five seconds -- with the integers relaxed as well, so it is the
|
||||||
|
# LP and not the branch and bound. The same model with uniform weights (ratio 41) solves in
|
||||||
|
# 3.8 s, five times out of five, matching HiGHS to 7e-5. With CVaR as a *constraint* the
|
||||||
|
# weights sit in a matrix row instead, where cuOpt's scaling copes, and it solves either way.
|
||||||
|
weighted_cvar_objective = req.objective == "min_cvar" and spec.probabilities is not None
|
||||||
|
if weighted_cvar_objective and req.solver == "auto":
|
||||||
|
info["solver_routing"] = "highs: CVaR objective with probability-weighted scenarios"
|
||||||
|
solve = self._solve_fn(req, prefer_local=weighted_cvar_objective, deadline=deadline)
|
||||||
|
if req.uses_row_generation(spec.n_scenarios):
|
||||||
|
sol, problem, rg = solve_by_row_generation(spec, solve, deadline=deadline)
|
||||||
|
return sol, problem, {**info, "row_generation": rg.as_dict()}
|
||||||
|
problem = MeanCVaRProblem(spec)
|
||||||
|
return solve(problem.build()), problem, info
|
||||||
|
|
||||||
|
# -- main entry point -------------------------------------------------------------------------
|
||||||
|
def optimize(self, req: OptimizationRequest) -> OptimizationResult:
|
||||||
|
timings: dict = {}
|
||||||
|
t0 = time.monotonic()
|
||||||
|
symbols = self.candidate_symbols(req)
|
||||||
|
mu_ann, er_desc, _bl = self.expected_returns(req, symbols)
|
||||||
|
timings["expected_returns_s"] = round(time.monotonic() - t0, 3)
|
||||||
|
extra_info: dict = {}
|
||||||
|
warnings: list[str] = []
|
||||||
|
|
||||||
|
if req.scenario_model == "gaussian_exact":
|
||||||
|
t1 = time.monotonic()
|
||||||
|
spec, scen_desc = self.build_gaussian_spec(req, symbols, mu_ann)
|
||||||
|
warnings.extend(self._sampling_warnings(len(symbols), 10 ** 9, scen_desc["covariance_model"]))
|
||||||
|
timings["model_s"] = round(time.monotonic() - t1, 3)
|
||||||
|
t3 = time.monotonic()
|
||||||
|
# the cut MILPs are tiny (N integers, a few dozen rows): HiGHS locally unless cuOpt is requested
|
||||||
|
sol, problem, oa = solve_gaussian_exact(spec, self._solve_fn(req, prefer_local=True,
|
||||||
|
deadline=self._deadline(req)))
|
||||||
|
extra_info["outer_approximation"] = oa.as_dict()
|
||||||
|
timings["solve_s"] = round(time.monotonic() - t3, 3)
|
||||||
|
exp_simple = spec.gaussian_mean
|
||||||
|
scen_hash = None
|
||||||
|
else:
|
||||||
|
t1 = time.monotonic()
|
||||||
|
S = req.scenario_count(len(symbols))
|
||||||
|
mode = req.tilt_mode(S)
|
||||||
|
tilt = None
|
||||||
|
if mode == "adaptive":
|
||||||
|
# stage 1: an untilted (moment-matched) set gives the loss direction for the tilt
|
||||||
|
scen1 = self.scenarios(req, symbols, mu_ann, n_scenarios=min(S, STAGE1_SCENARIOS), tilt_mode="off")
|
||||||
|
spec1 = self.build_spec(req, symbols, scen1)
|
||||||
|
sol1, problem1, _ = self._solve_scenarios(req, spec1)
|
||||||
|
if sol1.ok:
|
||||||
|
exposure = problem1.decode(sol1.x)["value"]
|
||||||
|
if exposure @ spec1.prices > 0:
|
||||||
|
tilt = TiltSpec.at_confidence(exposure, req.confidence, req.is_mixture * 2)
|
||||||
|
extra_info["importance_sampling"] = {"stage1_scenarios": scen1.n_scenarios,
|
||||||
|
"stage1_status": sol1.status}
|
||||||
|
if tilt is None:
|
||||||
|
mode = "off"
|
||||||
|
warnings.append("importance sampling skipped: no first-pass portfolio")
|
||||||
|
scen = self.scenarios(req, symbols, mu_ann, tilt=tilt, tilt_mode=mode)
|
||||||
|
warnings.extend(self._sampling_warnings(len(symbols), scen.n_scenarios, scen.covariance_model))
|
||||||
|
timings["scenarios_s"] = round(time.monotonic() - t1, 3)
|
||||||
|
t2 = time.monotonic()
|
||||||
|
spec = self.build_spec(req, symbols, scen)
|
||||||
|
timings["build_s"] = round(time.monotonic() - t2, 3)
|
||||||
|
t3 = time.monotonic()
|
||||||
|
sol, problem, rg = self._solve_scenarios(req, spec)
|
||||||
|
extra_info.update(rg)
|
||||||
|
timings["solve_s"] = round(time.monotonic() - t3, 3)
|
||||||
|
scen_desc = scen.describe()
|
||||||
|
exp_simple = scen.expected_simple
|
||||||
|
scen_hash = scen.hash
|
||||||
|
extra_info["scenario_cache"] = {"key": scen.extra["key"]["id"], "reused": bool(scen.extra.get("cached"))}
|
||||||
|
timings["total_s"] = round(time.monotonic() - t0, 3)
|
||||||
|
key = None if req.scenario_model == "gaussian_exact" else scen.extra["key"]
|
||||||
|
return self._result(req, symbols, spec, problem, sol, scen_desc, scen_hash, exp_simple, er_desc, mu_ann,
|
||||||
|
timings, extra_info, warnings, key)
|
||||||
|
|
||||||
|
def _result(self, req, symbols, spec, problem, sol, scen_desc, scen_hash, exp_simple, er_desc, mu_ann, timings,
|
||||||
|
extra_info, warnings, scenario_key=None) -> OptimizationResult:
|
||||||
|
h = req.horizon_periods
|
||||||
|
warnings = list(warnings)
|
||||||
|
if sol.status == "FeasibleFound":
|
||||||
|
# FeasibleFound can come from the solver's own limit *or* from an iteration cap in the
|
||||||
|
# outer approximation / row generation, which is not a time limit at all
|
||||||
|
oa, rg = extra_info.get("outer_approximation") or {}, extra_info.get("row_generation") or {}
|
||||||
|
if oa and not oa.get("converged"):
|
||||||
|
why = f"the Gaussian outer approximation stopped after {oa.get('iterations')} cuts"
|
||||||
|
elif rg and not rg.get("converged"):
|
||||||
|
why = f"the tail-row generation stopped after {rg.get('iterations')} passes"
|
||||||
|
else:
|
||||||
|
why = "the solver stopped at its limit"
|
||||||
|
warnings.append(f"{why}; solution feasible but not proven optimal (gap {sol.mip_gap})")
|
||||||
|
if "fallback" in sol.message:
|
||||||
|
warnings.append(sol.message)
|
||||||
|
if extra_info.get("solver_routing"):
|
||||||
|
warnings.append(
|
||||||
|
"solved on HiGHS (CPU) although cuOpt is reachable: cuOpt 26.08 aborts when the CVaR "
|
||||||
|
"is the objective *and* the scenarios are probability-weighted, because importance "
|
||||||
|
"sampling spreads the objective coefficients over six orders of magnitude. Set "
|
||||||
|
"importance_sampling='off' to use the GPU with this objective, or keep the weights "
|
||||||
|
"and the CPU - HiGHS solves it in a couple of seconds either way")
|
||||||
|
milp_desc = {"n_vars": problem.n_vars, "n_scenarios": spec.n_scenarios, "gaussian": problem.gaussian,
|
||||||
|
"weighted_scenarios": spec.probabilities is not None}
|
||||||
|
base_kwargs = dict(
|
||||||
|
status=sol.status, solver=sol.solver, mode=req.mode, as_of=scen_desc.get("as_of", ""), wealth=spec.wealth,
|
||||||
|
cash_start=req.cash, cvar_limit=spec.cvar_limit, confidence=req.confidence, horizon=req.horizon,
|
||||||
|
horizon_periods=h, n_candidates=len(symbols),
|
||||||
|
model={"expected_returns": er_desc, "scenarios": scen_desc, "milp": milp_desc,
|
||||||
|
"constraints": asdict(req.constraints), "costs": asdict(req.costs), "objective": req.objective,
|
||||||
|
"sampling": {"sampling": req.sampling, "moment_match": req.moment_match,
|
||||||
|
"importance_sampling": req.importance_sampling, "row_generation": req.row_generation,
|
||||||
|
"drift_convention": req.drift_convention, "risk_drift": req.risk_drift}},
|
||||||
|
solver_info={"status": sol.status, "solver": sol.solver, "solve_time_s": sol.solve_time,
|
||||||
|
"mip_gap": sol.mip_gap, "objective": sol.objective, "message": sol.message, **sol.raw,
|
||||||
|
**extra_info},
|
||||||
|
timings=timings, reproducible=sol.reproducible, scenario_hash=scen_hash, scenario_key=scenario_key,
|
||||||
|
before=(self.portfolio_figures(spec, spec.holdings, req.cash) if spec.rebalancing else None),
|
||||||
|
warnings=warnings,
|
||||||
|
# HiGHS' branch-and-bound is deterministic: identical inputs give identical shares. cuOpt (26.08)
|
||||||
|
# ignores gap settings over REST and its GPU heuristics stop within ~1e-4 of the optimum: identical
|
||||||
|
# objective, CVaR and expected P&L up to that gap, but near-tied share counts can differ.
|
||||||
|
reproducibility=("none" if not sol.reproducible else "exact" if sol.solver == "highs" else "within_gap"),
|
||||||
|
)
|
||||||
|
if not sol.ok:
|
||||||
|
warnings.append(f"no solution: {sol.status} {sol.message}".strip())
|
||||||
|
return OptimizationResult(cash_end=req.cash, invested=0.0, total_cost=0.0, turnover=0.0, expected_pnl=0.0,
|
||||||
|
expected_return=0.0, var=float("nan"), cvar=float("nan"), n_positions=0,
|
||||||
|
positions=[], sector_exposure={}, **base_kwargs)
|
||||||
|
d = problem.decode(sol.x)
|
||||||
|
if not problem.gaussian and spec.probabilities is not None and d["invested"] > 0:
|
||||||
|
alignment = self.tilt_alignment(spec, d["value"], self.market_exposure(symbols))
|
||||||
|
base_kwargs["solver_info"]["tilt_alignment"] = round(alignment, 3)
|
||||||
|
if np.isfinite(alignment) and alignment < TILT_ALIGNMENT_WARN:
|
||||||
|
warnings.append(
|
||||||
|
f"the solution's loss is barely correlated with the importance-sampling reference "
|
||||||
|
f"(alignment {alignment:.2f}); the tilt did not help here - consider "
|
||||||
|
f"importance_sampling='adaptive' or 'off'")
|
||||||
|
if spec.cvar_limit is not None and d["cvar"] > spec.cvar_limit * (1 + 1e-6):
|
||||||
|
rg = extra_info.get("row_generation") or {}
|
||||||
|
oa = extra_info.get("outer_approximation") or {}
|
||||||
|
over = d["cvar"] / spec.cvar_limit - 1
|
||||||
|
# enough precision for a small overshoot: "by 0.0%" reads as no breach at all
|
||||||
|
amount = f"{d['cvar'] - spec.cvar_limit:,.0f}" if over >= 5e-4 else f"{d['cvar'] - spec.cvar_limit:,.2f}"
|
||||||
|
# name the mechanism that left the gap, and the remedy that applies to *it*
|
||||||
|
if rg and not rg.get("converged"):
|
||||||
|
cause = (f": the tail-row generation stopped after {rg.get('iterations')} passes "
|
||||||
|
f"({rg.get('rows_final')} of {spec.n_scenarios} rows), so scenarios outside the "
|
||||||
|
f"working set are not covered - raise time_limit or set row_generation='off'")
|
||||||
|
elif oa and not oa.get("converged"):
|
||||||
|
cause = (f": the Gaussian outer approximation stopped after {oa.get('iterations')} cuts "
|
||||||
|
f"without converging, and it bounds the CVaR from below until it does - raise "
|
||||||
|
f"time_limit, or use scenario_model='gbm' for an exactly enforced limit")
|
||||||
|
else:
|
||||||
|
cause = (": the solver stopped within its MIP gap rather than at the true optimum - raise "
|
||||||
|
"time_limit, or lower n_scenarios / max_assets")
|
||||||
|
warnings.append(
|
||||||
|
f"the proposal BREACHES the CVaR limit by {amount} ({over:.3%}: {d['cvar']:,.0f} against "
|
||||||
|
f"{spec.cvar_limit:,.0f})" + cause)
|
||||||
|
# Minimising CVaR puts as little at risk as it is allowed to: cash has a CVaR of zero, so
|
||||||
|
# without a target return or a floor on the invested fraction the answer is trivially cash,
|
||||||
|
# and with a small target it is a sliver of the wealth. Both are correct and both look broken.
|
||||||
|
if sol.ok and req.objective == "min_cvar" and req.constraints.min_invested <= 0:
|
||||||
|
share = d["invested"] / spec.wealth if spec.wealth else 0.0
|
||||||
|
if share < 0.5:
|
||||||
|
target = "no target return" if not req.target_return else f"a target return of {req.target_return:.3%}"
|
||||||
|
warnings.append(
|
||||||
|
f"only {share:.1%} of the wealth is invested: minimising CVaR holds as little risk "
|
||||||
|
f"as the constraints allow, and cash has a CVaR of zero. With {target} and no "
|
||||||
|
f"min_invested floor this is the true optimum - raise target_return, or set "
|
||||||
|
f"constraints.min_invested (e.g. 0.95) to make it deploy the money")
|
||||||
|
if sol.ok and d["invested"] <= 0:
|
||||||
|
if req.objective == "min_cvar":
|
||||||
|
pass # already explained above, in more useful detail
|
||||||
|
elif req.objective == "utility":
|
||||||
|
# E[P&L] and CVaR are both money over the horizon, so lambda is dimensionless and
|
||||||
|
# its natural scale is E[P&L]/CVaR at the optimum -- of the order of 0.02 over a
|
||||||
|
# week. At lambda = 1 every franc of expected shortfall costs a franc of expected
|
||||||
|
# return and staying in cash is genuinely optimal, which is easy to mistake for a bug.
|
||||||
|
warnings.append(
|
||||||
|
f"nothing was invested: with risk_aversion={req.risk_aversion:g} the penalty "
|
||||||
|
f"{req.risk_aversion:g} x CVaR exceeds the expected P&L of every feasible "
|
||||||
|
f"portfolio, so cash maximises the utility. Over {h} periods the trade-off is of "
|
||||||
|
f"the order of 0.02; lower risk_aversion, or use objective='max_return' with a "
|
||||||
|
f"cvar_limit")
|
||||||
|
elif req.objective != "min_cvar":
|
||||||
|
warnings.append("nothing was invested: the constraints leave no position worth taking")
|
||||||
|
if spec.cvar_limit is not None and d["invested"] > 0:
|
||||||
|
if d["cvar"] <= 0:
|
||||||
|
warnings.append(
|
||||||
|
f"the {req.confidence:.0%} CVaR of the proposal is a *gain* ({-d['cvar']:,.0f}): over "
|
||||||
|
f"{h} periods the expected return ({d['expected_pnl'] / spec.wealth:.1%} of wealth) is larger "
|
||||||
|
f"than the tail of the return distribution, so the risk limit constrains nothing and the "
|
||||||
|
f"portfolio is driven by the return forecast and the weight caps alone - use "
|
||||||
|
f"risk_drift='zero', a shorter horizon, or a smaller limit")
|
||||||
|
elif d["cvar"] < 0.25 * spec.cvar_limit:
|
||||||
|
warnings.append(
|
||||||
|
f"the CVaR limit is far from binding ({d['cvar']:,.0f} against a limit of "
|
||||||
|
f"{spec.cvar_limit:,.0f}): the proposal is set by the weight and sector caps, not by risk")
|
||||||
|
held_mu = np.array([mu_ann.get(s, 0.0) for i, s in enumerate(symbols) if d["shares"][i] > 0])
|
||||||
|
if held_mu.size and np.nanmax(held_mu) > 0.60:
|
||||||
|
warnings.append(
|
||||||
|
f"the expected-return model gives up to {np.nanmax(held_mu):.0%} p.a. to a held stock "
|
||||||
|
f"({er_desc.get('type')}): an optimiser maximising expected return will concentrate in exactly "
|
||||||
|
f"those names - check the model before reading the proposal")
|
||||||
|
u = self.md.universe
|
||||||
|
positions = []
|
||||||
|
for i, s in enumerate(symbols):
|
||||||
|
if d["shares"][i] == 0 and d["trades"][i] == 0:
|
||||||
|
continue
|
||||||
|
positions.append({
|
||||||
|
"symbol": s, "name": u.at[s, "name"] if s in u.index else s, "sector": spec.sectors[i],
|
||||||
|
"currency": u.at[s, "listing_currency"] if s in u.index else None,
|
||||||
|
"price": float(spec.prices[i]), "shares": int(d["shares"][i]),
|
||||||
|
"shares_before": int(spec.holdings[i]), "trade": int(d["trades"][i]),
|
||||||
|
"value": float(d["value"][i]), "weight": float(d["weights"][i]),
|
||||||
|
"trade_value": float(d["trades"][i] * spec.prices[i]), "cost": float(d["cost"][i]),
|
||||||
|
"expected_return_ann": float(mu_ann.get(s, np.nan)),
|
||||||
|
"expected_return_horizon": float(exp_simple[i]),
|
||||||
|
})
|
||||||
|
positions.sort(key=lambda p: -p["value"])
|
||||||
|
sector_exposure = pd.Series(d["value"]).groupby(pd.Series(spec.sectors)).sum() / spec.wealth
|
||||||
|
return OptimizationResult(
|
||||||
|
cash_end=d["cash_end"], invested=d["invested"], total_cost=d["total_cost"], turnover=d["turnover"],
|
||||||
|
expected_pnl=d["expected_pnl"], expected_return=d["expected_pnl"] / spec.wealth, var=d["var"],
|
||||||
|
cvar=d["cvar"], n_positions=int((d["shares"] > 0).sum()), positions=positions,
|
||||||
|
sector_exposure={k: float(v) for k, v in sector_exposure.round(6).items() if v > 0}, **base_kwargs,
|
||||||
|
)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def tilt_alignment(spec: ProblemSpec, exposure: np.ndarray, reference: np.ndarray) -> float:
|
||||||
|
"""Correlation between the losses of the solution and of the tilt's reference portfolio.
|
||||||
|
|
||||||
|
The importance-sampling tilt enriches the tail *of the reference portfolio*; a solution
|
||||||
|
whose loss is highly correlated with it gets the full variance reduction (measured: 2.3x
|
||||||
|
to 5.3x at beta = 0.99 for alignments of 0.9 to 1.0), one that is uncorrelated gets none
|
||||||
|
and one that is anti-correlated is estimated slightly *less* precisely than without the
|
||||||
|
tilt. Long-only portfolios under single-stock and sector caps are market-like in practice
|
||||||
|
(alignment around 0.9), but the figure is reported so that the exception is visible.
|
||||||
|
"""
|
||||||
|
w = spec.weights # both arguments are money exposures per asset (any common scale)
|
||||||
|
a = spec.scenarios @ np.asarray(exposure, dtype=float)
|
||||||
|
b = spec.scenarios @ np.asarray(reference, dtype=float)
|
||||||
|
a = a - w @ a
|
||||||
|
b = b - w @ b
|
||||||
|
denom = np.sqrt((w @ (a * a)) * (w @ (b * b)))
|
||||||
|
return float((w @ (a * b)) / denom) if denom > 0 else float("nan")
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def portfolio_figures(spec: ProblemSpec, shares: np.ndarray, cash: float, total_cost: float = 0.0) -> dict:
|
||||||
|
"""Expected P&L, VaR, CVaR and exposures of a share portfolio under this spec's model.
|
||||||
|
|
||||||
|
The same quantities the solution reports, so that the starting portfolio of a rebalancing
|
||||||
|
can be compared with the proposal on identical scenarios (or, in Gaussian mode, on the
|
||||||
|
identical normal model).
|
||||||
|
"""
|
||||||
|
shares = np.asarray(shares, dtype=float)
|
||||||
|
value = shares * spec.prices
|
||||||
|
invested = float(value.sum())
|
||||||
|
expected_pnl = float(spec.mean_returns @ value) - total_cost
|
||||||
|
if spec.is_gaussian:
|
||||||
|
sigma = float(np.sqrt(max(value @ spec.gaussian_cov @ value, 0.0)))
|
||||||
|
z = float(norm.ppf(spec.confidence))
|
||||||
|
var_m, cvar_m = -expected_pnl + z * sigma, -expected_pnl + cvar_multiplier(spec.confidence) * sigma
|
||||||
|
else:
|
||||||
|
pnl = spec.scenarios @ value - total_cost
|
||||||
|
var_m, cvar_m = scenario_cvar(-pnl, spec.confidence, spec.probabilities)
|
||||||
|
sector = (pd.Series(value).groupby(pd.Series(spec.sectors)).sum() / spec.wealth).round(6)
|
||||||
|
return {
|
||||||
|
"invested": invested, "cash": float(cash), "wealth": float(cash + invested),
|
||||||
|
"n_positions": int((shares > 0).sum()), "expected_pnl": expected_pnl,
|
||||||
|
"expected_return": expected_pnl / spec.wealth if spec.wealth else 0.0,
|
||||||
|
"var": float(var_m), "cvar": float(cvar_m),
|
||||||
|
"cvar_fraction": float(cvar_m) / spec.wealth if spec.wealth else 0.0,
|
||||||
|
"weights": {s: float(value[i] / spec.wealth) for i, s in enumerate(spec.symbols) if value[i] != 0},
|
||||||
|
"sector_exposure": {k: float(v) for k, v in sector.items() if v > 0},
|
||||||
|
}
|
||||||
|
|
||||||
|
# -- verification helper: risk of an arbitrary share portfolio on the same scenario set -----------
|
||||||
|
@staticmethod
|
||||||
|
def portfolio_risk_on_scenarios(spec: ProblemSpec, shares: np.ndarray, confidence: float) -> dict:
|
||||||
|
pnl = spec.scenarios @ (np.asarray(shares) * spec.prices)
|
||||||
|
var_m, cvar_m = scenario_cvar(-pnl, confidence, spec.probabilities)
|
||||||
|
w = spec.weights
|
||||||
|
return {"expected_pnl": float((w @ pnl) / w.sum()), "var": var_m, "cvar": cvar_m}
|
||||||
317
quantcore/optimization/solvers.py
Normal file
317
quantcore/optimization/solvers.py
Normal file
@ -0,0 +1,317 @@
|
|||||||
|
"""MILP solvers behind one interface: NVIDIA cuOpt (REST, on the DGX Spark) and HiGHS (scipy).
|
||||||
|
|
||||||
|
The cuOpt server solves LP/MILP from a JSON body with the constraint matrix in CSR form and
|
||||||
|
answers asynchronously (submit -> poll). :class:`CuOptSolver` wraps this with timeouts, retry
|
||||||
|
with back-off on connection errors and a circuit breaker; :func:`solve_with_fallback` uses
|
||||||
|
the local HiGHS solver when cuOpt is unreachable (deterministic analysis must keep working
|
||||||
|
when an external dependency is down -- and the result says which solver produced it).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import requests
|
||||||
|
|
||||||
|
from quantcore.config import Settings
|
||||||
|
from quantcore.data.yahoo import CircuitBreaker, CircuitOpenError
|
||||||
|
from quantcore.optimization.problem import MILPModel
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
_HEALTH_CACHE: dict[str, tuple[float, bool]] = {} # base_url -> (checked_at, reachable)
|
||||||
|
|
||||||
|
CUOPT_HEADERS = {"CLIENT-VERSION": "custom", "Accept": "application/json", "Content-Type": "application/json"}
|
||||||
|
OPTIMAL_STATUSES = {"Optimal", "FeasibleFound"}
|
||||||
|
|
||||||
|
# cuOpt 26.08 aborts on some of these models rather than solving them, and which of its LP methods
|
||||||
|
# copes depends on the model. Measured on a 4 137-variable / 4 106-row mean-CVaR MILP (40 candidates,
|
||||||
|
# 4 096 scenarios, beta = 99 %): with CVaR in the *objective* (objective='min_cvar') the default
|
||||||
|
# concurrent method, the dual simplex (2) and method 3 all abort within ~5 s, while method 1 (PDLP)
|
||||||
|
# returns the optimum in 4.9 s, matching HiGHS to 7e-5. With CVaR as a *constraint* the default
|
||||||
|
# solves it. So an abort is retried down this ladder before the caller is told cuOpt could not do it.
|
||||||
|
CUOPT_RETRY_METHODS = (1, 2)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class SolverResult:
|
||||||
|
status: str # Optimal | FeasibleFound | Infeasible | Unbounded | TimeLimit | Error
|
||||||
|
x: np.ndarray | None
|
||||||
|
objective: float | None
|
||||||
|
solver: str
|
||||||
|
solve_time: float
|
||||||
|
mip_gap: float | None = None
|
||||||
|
message: str = ""
|
||||||
|
raw: dict = field(default_factory=dict)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def ok(self) -> bool:
|
||||||
|
return self.x is not None and self.status in OPTIMAL_STATUSES
|
||||||
|
|
||||||
|
@property
|
||||||
|
def reproducible(self) -> bool:
|
||||||
|
"""True when optimality was proven within the MIP gap: with the same model (same scenario
|
||||||
|
hash) any solver run returns the same objective value, and the same shares up to exact ties.
|
||||||
|
A ``FeasibleFound`` (time limit) result depends on the solver's timing and is not reproducible."""
|
||||||
|
return self.ok and self.status == "Optimal"
|
||||||
|
|
||||||
|
|
||||||
|
def _bound(v: float) -> float | str:
|
||||||
|
if np.isposinf(v):
|
||||||
|
return "inf"
|
||||||
|
if np.isneginf(v):
|
||||||
|
return "ninf"
|
||||||
|
return float(v)
|
||||||
|
|
||||||
|
|
||||||
|
def cuopt_solver_config(settings: Settings, time_limit: float, extra_config: dict | None = None) -> dict:
|
||||||
|
"""cuOpt ``solver_config``: time limit, MIP gaps (reproducibility), threads, no console log."""
|
||||||
|
cfg = {"time_limit": float(time_limit), "log_to_console": False,
|
||||||
|
"tolerances": {"mip_relative_gap": settings.mip_rel_gap, "mip_absolute_gap": settings.mip_abs_gap}}
|
||||||
|
if settings.cuopt_threads:
|
||||||
|
cfg["num_cpu_threads"] = int(settings.cuopt_threads)
|
||||||
|
if extra_config:
|
||||||
|
tol = {**cfg["tolerances"], **(extra_config.get("tolerances") or {})}
|
||||||
|
cfg = {**cfg, **extra_config, "tolerances": tol}
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
def to_cuopt_payload(model: MILPModel, time_limit: float, extra_config: dict | None = None,
|
||||||
|
settings: Settings | None = None) -> dict:
|
||||||
|
"""The cuOpt LP/MILP request body for a :class:`MILPModel`."""
|
||||||
|
A = model.A.tocsr()
|
||||||
|
payload = {
|
||||||
|
"csr_constraint_matrix": {
|
||||||
|
"offsets": A.indptr.tolist(), "indices": A.indices.tolist(), "values": A.data.tolist(),
|
||||||
|
},
|
||||||
|
"constraint_bounds": {
|
||||||
|
"upper_bounds": [_bound(v) for v in model.row_ub], "lower_bounds": [_bound(v) for v in model.row_lb],
|
||||||
|
},
|
||||||
|
"objective_data": {"coefficients": model.c.tolist(), "scalability_factor": 1.0, "offset": model.offset},
|
||||||
|
"variable_bounds": {
|
||||||
|
"upper_bounds": [_bound(v) for v in model.var_ub], "lower_bounds": [_bound(v) for v in model.var_lb],
|
||||||
|
},
|
||||||
|
"variable_types": list(model.var_types),
|
||||||
|
"variable_names": list(model.var_names),
|
||||||
|
"maximize": bool(model.maximize),
|
||||||
|
"solver_config": (cuopt_solver_config(settings, time_limit, extra_config) if settings is not None
|
||||||
|
else {"time_limit": float(time_limit), **(extra_config or {})}),
|
||||||
|
}
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def encode_payload(payload: dict, use_msgpack: bool) -> tuple[bytes, str]:
|
||||||
|
"""Serialise the request body: msgpack (about half the size of JSON, faster to parse) when
|
||||||
|
available and enabled, JSON otherwise."""
|
||||||
|
if use_msgpack:
|
||||||
|
try:
|
||||||
|
import msgpack
|
||||||
|
|
||||||
|
return msgpack.packb(payload, use_bin_type=True), "application/vnd.msgpack"
|
||||||
|
except ImportError: # pragma: no cover
|
||||||
|
pass
|
||||||
|
import json
|
||||||
|
|
||||||
|
return json.dumps(payload).encode(), "application/json"
|
||||||
|
|
||||||
|
|
||||||
|
class CuOptSolver:
|
||||||
|
"""Client of the cuOpt self-hosted server (``/cuopt/request`` + ``/cuopt/solution/{id}``)."""
|
||||||
|
|
||||||
|
name = "cuopt"
|
||||||
|
|
||||||
|
def __init__(self, settings: Settings, breaker: CircuitBreaker | None = None):
|
||||||
|
self.settings = settings
|
||||||
|
self.base_url = settings.cuopt_url.rstrip("/")
|
||||||
|
self.breaker = breaker or CircuitBreaker(threshold=3, cooldown=120.0, name="cuOpt")
|
||||||
|
|
||||||
|
# -- health ----------------------------------------------------------------------------------
|
||||||
|
def reachable(self, ttl: float = 30.0) -> bool:
|
||||||
|
"""Is the server up? Cached for ``ttl`` seconds (a solve by row generation asks repeatedly).
|
||||||
|
|
||||||
|
Without this an unreachable server costs the connect timeout on every attempt before the
|
||||||
|
HiGHS fallback takes over -- minutes of apparent hanging for ``solver='auto'``.
|
||||||
|
"""
|
||||||
|
now = time.monotonic()
|
||||||
|
hit = _HEALTH_CACHE.get(self.base_url)
|
||||||
|
if hit is not None and now - hit[0] < ttl:
|
||||||
|
return hit[1]
|
||||||
|
ok = bool(self.health(timeout=2.0).get("reachable"))
|
||||||
|
_HEALTH_CACHE[self.base_url] = (now, ok)
|
||||||
|
return ok
|
||||||
|
|
||||||
|
def health(self, timeout: float = 5.0) -> dict:
|
||||||
|
try:
|
||||||
|
r = requests.get(f"{self.base_url}/cuopt/health", headers=CUOPT_HEADERS, timeout=timeout)
|
||||||
|
r.raise_for_status()
|
||||||
|
return {"reachable": True, **r.json()}
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
return {"reachable": False, "error": str(exc)}
|
||||||
|
|
||||||
|
# -- solve -------------------------------------------------------------------------------------
|
||||||
|
def _post_with_retry(self, payload: dict) -> str:
|
||||||
|
body, content_type = encode_payload(payload, self.settings.cuopt_msgpack)
|
||||||
|
headers = {**CUOPT_HEADERS, "Content-Type": content_type}
|
||||||
|
# the POST blocks until the server has read and parsed the whole model, so the timeout has
|
||||||
|
# to scale with the payload: a client-side timeout here surfaces as an "aborted" request
|
||||||
|
# (connect, read): a big body needs a long *read* timeout, never a long connect wait -- and
|
||||||
|
# cuOpt serves one request at a time, so a busy server refuses the connection rather than stalling
|
||||||
|
# one scalar timeout: urllib3 governs the body write with the connect timeout, so a short
|
||||||
|
# connect timeout aborts large uploads mid-flight
|
||||||
|
timeout = self.settings.cuopt_request_timeout + self.settings.cuopt_upload_seconds_per_mb * len(body) / 1e6
|
||||||
|
delay = 1.0
|
||||||
|
last: Exception | None = None
|
||||||
|
for attempt in range(1, 4):
|
||||||
|
# a fresh connection per attempt: a timed-out POST leaves a partially written body in a
|
||||||
|
# pooled connection, and the retry would append a second body -- the server then reports
|
||||||
|
# "unpack received extra data" (422) or aborts the job
|
||||||
|
with requests.Session() as session:
|
||||||
|
try:
|
||||||
|
r = session.post(f"{self.base_url}/cuopt/request", data=body,
|
||||||
|
headers={**headers, "Connection": "close"}, timeout=timeout)
|
||||||
|
if r.status_code in (400, 422): # schema or decoding error: no point retrying
|
||||||
|
raise ValueError(f"cuOpt rejected the model ({r.status_code}): {r.text[:500]}")
|
||||||
|
r.raise_for_status()
|
||||||
|
return r.json()["reqId"]
|
||||||
|
except (requests.ConnectionError, requests.Timeout) as exc:
|
||||||
|
last = exc
|
||||||
|
log.warning("cuOpt submit failed (attempt %d/3): %s", attempt, exc)
|
||||||
|
time.sleep(delay)
|
||||||
|
delay *= 2
|
||||||
|
raise ConnectionError(
|
||||||
|
f"cuOpt unreachable at {self.base_url} (payload {len(body) / 1e6:.1f} MB, timeout "
|
||||||
|
f"{timeout:.0f} s per attempt)") from last
|
||||||
|
|
||||||
|
def solve(self, model: MILPModel, time_limit: float | None = None, extra_config: dict | None = None,
|
||||||
|
_retry: bool = True) -> SolverResult:
|
||||||
|
"""Submit, poll, parse. An aborted solve is retried down :data:`CUOPT_RETRY_METHODS`."""
|
||||||
|
self.breaker.check()
|
||||||
|
time_limit = time_limit or self.settings.cuopt_time_limit
|
||||||
|
payload = to_cuopt_payload(model, time_limit, extra_config, self.settings)
|
||||||
|
t0 = time.monotonic()
|
||||||
|
try:
|
||||||
|
req_id = self._post_with_retry(payload)
|
||||||
|
except (ConnectionError, ValueError):
|
||||||
|
self.breaker.failure()
|
||||||
|
raise
|
||||||
|
deadline = t0 + self.settings.cuopt_timeout
|
||||||
|
url = f"{self.base_url}/cuopt/solution/{req_id}"
|
||||||
|
try:
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
try:
|
||||||
|
r = requests.get(url, headers=CUOPT_HEADERS, timeout=self.settings.cuopt_request_timeout)
|
||||||
|
except (requests.ConnectionError, requests.Timeout) as exc:
|
||||||
|
# the server went away mid-solve (e.g. its solver process died and took the web
|
||||||
|
# server with it): raise the builtin ConnectionError that solve_with_fallback
|
||||||
|
# turns into a HiGHS solve -- requests' own class is not a subclass of it
|
||||||
|
self.breaker.failure()
|
||||||
|
raise ConnectionError(f"cuOpt connection lost during the solve at {self.base_url}: "
|
||||||
|
f"{str(exc)[:120]}") from exc
|
||||||
|
body = r.json()
|
||||||
|
if body.get("error_result"):
|
||||||
|
message = str(body.get("error"))
|
||||||
|
if _retry and "abort" in message.lower():
|
||||||
|
for method in CUOPT_RETRY_METHODS:
|
||||||
|
if (extra_config or {}).get("method") == method:
|
||||||
|
continue
|
||||||
|
log.warning("cuOpt aborted (%s) - retrying with method=%d", message[:60], method)
|
||||||
|
retried = self.solve(model, time_limit, {**(extra_config or {}), "method": method},
|
||||||
|
_retry=False)
|
||||||
|
if retried.ok:
|
||||||
|
retried.message = (f"{retried.message} (cuOpt aborted with its default method; "
|
||||||
|
f"solved with method={method})").strip()
|
||||||
|
return retried
|
||||||
|
message = (f"{message} - and every method in {CUOPT_RETRY_METHODS} aborted too")
|
||||||
|
self.breaker.failure()
|
||||||
|
return SolverResult("Error", None, None, self.name, time.monotonic() - t0, message=message,
|
||||||
|
raw=body)
|
||||||
|
if "response" in body:
|
||||||
|
self.breaker.success()
|
||||||
|
return self._parse(body["response"]["solver_response"], time.monotonic() - t0)
|
||||||
|
time.sleep(self.settings.cuopt_poll_interval)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
requests.delete(url, headers=CUOPT_HEADERS, timeout=10)
|
||||||
|
except requests.RequestException:
|
||||||
|
pass
|
||||||
|
return SolverResult("TimeLimit", None, None, self.name, time.monotonic() - t0, message="client-side timeout")
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _parse(resp: dict, elapsed: float) -> SolverResult:
|
||||||
|
status = str(resp.get("status", "Unknown"))
|
||||||
|
sol = resp.get("solution") or {}
|
||||||
|
x = sol.get("primal_solution")
|
||||||
|
stats = sol.get("milp_statistics") or {}
|
||||||
|
return SolverResult(
|
||||||
|
status=status, x=None if x is None else np.asarray(x, dtype=float),
|
||||||
|
objective=sol.get("primal_objective"), solver="cuopt", solve_time=float(sol.get("solver_time", elapsed)),
|
||||||
|
mip_gap=stats.get("mip_gap"), raw={"status": status, "milp_statistics": stats,
|
||||||
|
"problem_category": sol.get("problem_category")},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class HighsSolver:
|
||||||
|
"""Local MILP solver (HiGHS via ``scipy.optimize.milp``) -- the fallback and the test oracle."""
|
||||||
|
|
||||||
|
name = "highs"
|
||||||
|
|
||||||
|
def __init__(self, settings: Settings | None = None):
|
||||||
|
self.settings = settings
|
||||||
|
|
||||||
|
def solve(self, model: MILPModel, time_limit: float | None = None, mip_gap: float | None = None) -> SolverResult:
|
||||||
|
from scipy.optimize import Bounds, LinearConstraint, milp
|
||||||
|
|
||||||
|
if mip_gap is None:
|
||||||
|
mip_gap = self.settings.mip_rel_gap if self.settings is not None else 1e-4
|
||||||
|
t0 = time.monotonic()
|
||||||
|
c = -model.c if model.maximize else model.c
|
||||||
|
integrality = np.array([1 if t == "I" else 0 for t in model.var_types])
|
||||||
|
cons = LinearConstraint(model.A, model.row_lb, model.row_ub)
|
||||||
|
bounds = Bounds(model.var_lb, model.var_ub)
|
||||||
|
options = {"disp": False, "mip_rel_gap": mip_gap}
|
||||||
|
if time_limit:
|
||||||
|
options["time_limit"] = float(time_limit)
|
||||||
|
res = milp(c, constraints=cons, integrality=integrality, bounds=bounds, options=options)
|
||||||
|
elapsed = time.monotonic() - t0
|
||||||
|
status_map = {0: "Optimal", 1: "IterationLimit", 2: "Infeasible", 3: "Unbounded", 4: "Error"}
|
||||||
|
status = status_map.get(res.status, "Unknown")
|
||||||
|
if res.status == 1 and res.x is not None:
|
||||||
|
status = "FeasibleFound"
|
||||||
|
x = None if res.x is None else np.asarray(res.x, dtype=float)
|
||||||
|
obj = None if x is None else float(model.c @ x + model.offset)
|
||||||
|
return SolverResult(status, x, obj, self.name, elapsed, mip_gap=getattr(res, "mip_gap", None),
|
||||||
|
message=str(res.message), raw={"status_code": int(res.status)})
|
||||||
|
|
||||||
|
|
||||||
|
def solve_with_fallback(model: MILPModel, settings: Settings, solver: str = "auto",
|
||||||
|
time_limit: float | None = None) -> SolverResult:
|
||||||
|
"""``solver``: ``'cuopt'``, ``'highs'`` or ``'auto'`` (cuOpt, HiGHS if cuOpt is unreachable)."""
|
||||||
|
if solver == "highs":
|
||||||
|
return HighsSolver(settings).solve(model, time_limit)
|
||||||
|
cu = CuOptSolver(settings)
|
||||||
|
if solver == "auto" and settings.solver_fallback and not cu.reachable():
|
||||||
|
log.info("cuOpt is not reachable at %s - solving with HiGHS", settings.cuopt_url)
|
||||||
|
result = HighsSolver(settings).solve(model, time_limit)
|
||||||
|
result.message = (result.message + " (fallback: cuOpt unreachable)").strip()
|
||||||
|
return result
|
||||||
|
reason = "cuOpt unavailable"
|
||||||
|
try:
|
||||||
|
result = cu.solve(model, time_limit)
|
||||||
|
if result.status != "Error" or solver == "cuopt":
|
||||||
|
return result
|
||||||
|
log.warning("cuOpt returned an error (%s)", result.message)
|
||||||
|
# the server is up and answered -- say so, or the reader sees a green health pill next to a
|
||||||
|
# result that claims cuOpt was "unavailable" and concludes the GPU is being ignored
|
||||||
|
reason = ("cuOpt is reachable but aborted on this model" if "abort" in result.message.lower()
|
||||||
|
else f"cuOpt returned an error: {result.message[:80]}")
|
||||||
|
except (ConnectionError, CircuitOpenError, ValueError) as exc:
|
||||||
|
if solver == "cuopt" or not settings.solver_fallback:
|
||||||
|
raise
|
||||||
|
log.warning("cuOpt unavailable (%s) - falling back to HiGHS", exc)
|
||||||
|
reason = f"cuOpt unreachable: {str(exc)[:80]}"
|
||||||
|
result = HighsSolver(settings).solve(model, time_limit)
|
||||||
|
result.message = f"{result.message} (fallback to HiGHS: {reason})".strip()
|
||||||
|
return result
|
||||||
18
quantcore/universe/__init__.py
Normal file
18
quantcore/universe/__init__.py
Normal file
@ -0,0 +1,18 @@
|
|||||||
|
"""Investment universe: candidate sources, Yahoo symbol mapping and the one-off ``SetupService``."""
|
||||||
|
|
||||||
|
from quantcore.universe.setup_service import SetupService, UniverseBuildReport
|
||||||
|
from quantcore.universe.sources import (
|
||||||
|
fetch_sp500_wikipedia,
|
||||||
|
fetch_stoxx600_ishares,
|
||||||
|
load_static_candidates,
|
||||||
|
)
|
||||||
|
from quantcore.universe.symbols import to_yahoo_symbol
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"SetupService",
|
||||||
|
"UniverseBuildReport",
|
||||||
|
"fetch_sp500_wikipedia",
|
||||||
|
"fetch_stoxx600_ishares",
|
||||||
|
"load_static_candidates",
|
||||||
|
"to_yahoo_symbol",
|
||||||
|
]
|
||||||
209
quantcore/universe/setup_service.py
Normal file
209
quantcore/universe/setup_service.py
Normal file
@ -0,0 +1,209 @@
|
|||||||
|
"""``SetupService`` -- builds the investment universe once and persists everything.
|
||||||
|
|
||||||
|
Steps (each is a method, so parts can be re-run):
|
||||||
|
|
||||||
|
1. **candidates**: S&P 500 (US) and STOXX Europe 600 (Europe) constituents, see
|
||||||
|
:mod:`quantcore.universe.sources`.
|
||||||
|
2. **metadata**: Yahoo ``Ticker.info`` for every candidate -- listing currency, exchange,
|
||||||
|
domicile, Yahoo sector/industry, market capitalisation, shares outstanding, first trade date.
|
||||||
|
3. **prices**: complete adjusted-close history since ``settings.history_start`` for all
|
||||||
|
candidates (so later universe changes need no re-download); FX rates to the base currency.
|
||||||
|
4. **selection**: one listing per company, rank by market capitalisation in the base currency,
|
||||||
|
keep the largest ``n_us`` / ``n_europe``, then drop stocks without prices since 2005
|
||||||
|
(``settings.required_history_from``) or with poor coverage -- which is why the final
|
||||||
|
universe has fewer than 1000 names.
|
||||||
|
5. **persist**: ``universe/universe.csv`` with ticker, company name, country, listing currency,
|
||||||
|
exchange, GICS sector, Yahoo sector/industry, market cap (local and base), first price date.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from quantcore.config import Settings
|
||||||
|
from quantcore.data.cleaning import clean_prices, longest_missing_run
|
||||||
|
from quantcore.data.fx import download_fx
|
||||||
|
from quantcore.data.store import DataStore
|
||||||
|
from quantcore.data.yahoo import YahooClient, apply_price_scale, coverage_since, first_valid_dates
|
||||||
|
from quantcore.universe.sources import EUROPE_COUNTRIES, fetch_candidates
|
||||||
|
from quantcore.universe.symbols import normalise_share_class_name
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
UNIVERSE_COLUMNS = [
|
||||||
|
"symbol", "name", "region", "country", "listing_currency", "exchange", "exchange_code", "gics_sector",
|
||||||
|
"gics_sub_industry", "yahoo_sector", "yahoo_industry", "market_cap", "market_cap_base",
|
||||||
|
"shares_outstanding", "last_price", "first_price_date", "price_coverage", "max_missing_run", "index_weight",
|
||||||
|
"source", "rank_in_region",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class UniverseBuildReport:
|
||||||
|
started_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat(timespec="seconds"))
|
||||||
|
n_candidates: int = 0
|
||||||
|
n_resolved: int = 0
|
||||||
|
n_with_prices: int = 0
|
||||||
|
n_selected: int = 0
|
||||||
|
n_universe: int = 0
|
||||||
|
by_region: dict = field(default_factory=dict)
|
||||||
|
by_country: dict = field(default_factory=dict)
|
||||||
|
by_sector: dict = field(default_factory=dict)
|
||||||
|
dropped_no_history: list[str] = field(default_factory=list)
|
||||||
|
dropped_unresolved: list[str] = field(default_factory=list)
|
||||||
|
first_price_date: str | None = None
|
||||||
|
last_price_date: str | None = None
|
||||||
|
finished_at: str | None = None
|
||||||
|
|
||||||
|
def as_dict(self) -> dict:
|
||||||
|
return self.__dict__.copy()
|
||||||
|
|
||||||
|
|
||||||
|
class SetupService:
|
||||||
|
"""One-off construction of the investment universe (idempotent; re-run to refresh)."""
|
||||||
|
|
||||||
|
def __init__(self, settings: Settings, store: DataStore | None = None, client: YahooClient | None = None):
|
||||||
|
self.settings = settings
|
||||||
|
self.store = store or DataStore(settings)
|
||||||
|
self.client = client or YahooClient(settings)
|
||||||
|
|
||||||
|
# -- step 1 --------------------------------------------------------------------------------
|
||||||
|
def candidates(self, allow_web: bool = True, max_per_region: int | None = None) -> pd.DataFrame:
|
||||||
|
us = fetch_candidates("US", allow_web=allow_web)
|
||||||
|
eu = fetch_candidates("Europe", allow_web=allow_web)
|
||||||
|
if max_per_region:
|
||||||
|
us, eu = us.head(max_per_region), eu.head(max_per_region)
|
||||||
|
df = pd.concat([us, eu], ignore_index=True).drop_duplicates(subset="symbol").reset_index(drop=True)
|
||||||
|
self.store.save_candidates(df)
|
||||||
|
log.info("candidates: %d US + %d Europe = %d", len(us), len(eu), len(df))
|
||||||
|
return df
|
||||||
|
|
||||||
|
# -- step 2 --------------------------------------------------------------------------------
|
||||||
|
def metadata(self, symbols: list[str]) -> pd.DataFrame:
|
||||||
|
def _progress(i, n):
|
||||||
|
log.info("metadata %d/%d", i, n)
|
||||||
|
|
||||||
|
meta = self.client.ticker_infos(symbols, progress=_progress)
|
||||||
|
self.store.save_metadata(meta)
|
||||||
|
return meta
|
||||||
|
|
||||||
|
# -- step 3 --------------------------------------------------------------------------------
|
||||||
|
def prices_and_fx(self, symbols: list[str], meta: pd.DataFrame) -> tuple[pd.DataFrame, pd.DataFrame]:
|
||||||
|
hist = self.client.download_history(symbols, start=self.settings.history_start)["Close"]
|
||||||
|
prices = clean_prices(apply_price_scale(hist, meta["price_scale"]))
|
||||||
|
self.store.save_prices(prices)
|
||||||
|
currencies = meta.loc[meta.index.isin(prices.columns), "currency"].dropna().unique().tolist()
|
||||||
|
fx = download_fx(self.client, currencies, self.settings.base_currency, self.settings.history_start)
|
||||||
|
self.store.save_fx(fx)
|
||||||
|
return prices, fx
|
||||||
|
|
||||||
|
# -- step 4 --------------------------------------------------------------------------------
|
||||||
|
def select(self, candidates: pd.DataFrame, meta: pd.DataFrame, prices: pd.DataFrame, fx: pd.DataFrame,
|
||||||
|
report: UniverseBuildReport | None = None) -> pd.DataFrame:
|
||||||
|
report = report or UniverseBuildReport()
|
||||||
|
s = self.settings
|
||||||
|
df = candidates.merge(meta, left_on="symbol", right_index=True, how="left")
|
||||||
|
resolved = df["quote_type"].eq("EQUITY") & df["currency"].notna()
|
||||||
|
report.dropped_unresolved = df.loc[~resolved, "symbol"].tolist()
|
||||||
|
df = df[resolved].copy()
|
||||||
|
report.n_resolved = len(df)
|
||||||
|
|
||||||
|
# price statistics
|
||||||
|
have = df["symbol"].isin(prices.columns)
|
||||||
|
df = df[have].copy()
|
||||||
|
report.n_with_prices = len(df)
|
||||||
|
first = first_valid_dates(prices[df["symbol"]])
|
||||||
|
cov = coverage_since(prices[df["symbol"]], pd.Timestamp(s.required_history_from))
|
||||||
|
gaps = longest_missing_run(prices[df["symbol"]], pd.Timestamp(s.required_history_from))
|
||||||
|
df["first_price_date"] = df["symbol"].map(first).map(lambda d: None if pd.isna(d) else pd.Timestamp(d).date().isoformat())
|
||||||
|
df["price_coverage"] = df["symbol"].map(cov).round(4)
|
||||||
|
df["max_missing_run"] = df["symbol"].map(gaps)
|
||||||
|
last_px = prices[df["symbol"]].ffill().iloc[-1]
|
||||||
|
df["last_price"] = df["last_price"].where(df["last_price"].notna(), df["symbol"].map(last_px))
|
||||||
|
|
||||||
|
# market capitalisation in the base currency
|
||||||
|
fx_last = fx.ffill().iloc[-1]
|
||||||
|
rate = df["currency"].map(fx_last).astype(float)
|
||||||
|
cap_local = df["market_cap"].astype(float)
|
||||||
|
est = df["shares_outstanding"].astype(float) * df["last_price"].astype(float)
|
||||||
|
cap_local = cap_local.where(cap_local.notna() & (cap_local > 0), est)
|
||||||
|
df["market_cap"] = cap_local
|
||||||
|
df["market_cap_base"] = cap_local * rate
|
||||||
|
df["listing_currency"] = df["currency"]
|
||||||
|
df["name"] = df["long_name"].where(df["long_name"].notna(), df["name"])
|
||||||
|
df["country"] = df["yahoo_country"].where(df["yahoo_country"].notna(), df["country"])
|
||||||
|
|
||||||
|
# Europe = EU + CH + NO + UK by domicile of the *source* (STOXX country), US = S&P 500 members
|
||||||
|
eu_mask = (df["region"] == "Europe") & ~candidates.set_index("symbol")["country"].reindex(df["symbol"]).isin(EUROPE_COUNTRIES).values
|
||||||
|
df = df[~eu_mask].copy()
|
||||||
|
|
||||||
|
# one listing per company: keep the largest market cap among share classes
|
||||||
|
df["company_key"] = df["region"] + "|" + df["name"].map(normalise_share_class_name)
|
||||||
|
df = df.sort_values("market_cap_base", ascending=False).drop_duplicates("company_key")
|
||||||
|
|
||||||
|
# rank within region and keep the largest N
|
||||||
|
df = df[df["market_cap_base"].notna() & (df["market_cap_base"] > 0)].copy()
|
||||||
|
df["rank_in_region"] = df.groupby("region")["market_cap_base"].rank(ascending=False, method="first").astype(int)
|
||||||
|
limit = df["region"].map({"US": s.n_us, "Europe": s.n_europe})
|
||||||
|
selected = df[df["rank_in_region"] <= limit].copy()
|
||||||
|
report.n_selected = len(selected)
|
||||||
|
|
||||||
|
# history filter: prices since 2005 and adequate coverage
|
||||||
|
has_history = (
|
||||||
|
(pd.to_datetime(selected["first_price_date"]) <= pd.Timestamp(s.required_history_from))
|
||||||
|
& (selected["price_coverage"] >= s.min_price_coverage)
|
||||||
|
& (selected["max_missing_run"] <= s.max_missing_run)
|
||||||
|
)
|
||||||
|
report.dropped_no_history = selected.loc[~has_history, "symbol"].tolist()
|
||||||
|
universe = selected[has_history].sort_values(["region", "rank_in_region"]).reset_index(drop=True)
|
||||||
|
for col in UNIVERSE_COLUMNS:
|
||||||
|
if col not in universe.columns:
|
||||||
|
universe[col] = np.nan
|
||||||
|
universe = universe[UNIVERSE_COLUMNS]
|
||||||
|
report.n_universe = len(universe)
|
||||||
|
report.by_region = universe["region"].value_counts().to_dict()
|
||||||
|
report.by_country = universe["country"].value_counts().to_dict()
|
||||||
|
report.by_sector = universe["gics_sector"].value_counts().to_dict()
|
||||||
|
return universe
|
||||||
|
|
||||||
|
# -- orchestration ----------------------------------------------------------------------------
|
||||||
|
def run(self, allow_web: bool = True, max_per_region: int | None = None) -> UniverseBuildReport:
|
||||||
|
"""Full setup: candidates -> metadata -> prices/FX -> selection -> persist."""
|
||||||
|
report = UniverseBuildReport()
|
||||||
|
self.store.ensure_dirs()
|
||||||
|
cands = self.candidates(allow_web=allow_web, max_per_region=max_per_region)
|
||||||
|
report.n_candidates = len(cands)
|
||||||
|
meta = self.metadata(cands["symbol"].tolist())
|
||||||
|
ok = meta.index[meta["quote_type"].eq("EQUITY")].tolist()
|
||||||
|
prices, fx = self.prices_and_fx(ok, meta)
|
||||||
|
universe = self.select(cands, meta, prices, fx, report)
|
||||||
|
self.store.save_universe(universe)
|
||||||
|
report.first_price_date = prices.index.min().date().isoformat()
|
||||||
|
report.last_price_date = prices.index.max().date().isoformat()
|
||||||
|
report.finished_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||||
|
self.store.update_manifest(
|
||||||
|
setup_at=report.finished_at, n_candidates=report.n_candidates, n_universe=report.n_universe,
|
||||||
|
by_region=report.by_region, first_price_date=report.first_price_date,
|
||||||
|
last_price_date=report.last_price_date, n_symbols=len(prices.columns),
|
||||||
|
base_currency=self.settings.base_currency, required_history_from=self.settings.required_history_from,
|
||||||
|
)
|
||||||
|
log.info("universe built: %d stocks (%s)", report.n_universe, report.by_region)
|
||||||
|
return report
|
||||||
|
|
||||||
|
def rebuild_selection(self) -> UniverseBuildReport:
|
||||||
|
"""Re-run only the selection step on stored data (e.g. after changing ``n_us``)."""
|
||||||
|
report = UniverseBuildReport()
|
||||||
|
cands = self.store.load_candidates()
|
||||||
|
meta = self.store.load_metadata()
|
||||||
|
prices = self.store.load_prices()
|
||||||
|
fx = self.store.load_fx()
|
||||||
|
report.n_candidates = len(cands)
|
||||||
|
universe = self.select(cands, meta, prices, fx, report)
|
||||||
|
self.store.save_universe(universe)
|
||||||
|
self.store.update_manifest(n_universe=report.n_universe, by_region=report.by_region)
|
||||||
|
return report
|
||||||
302
quantcore/universe/sources.py
Normal file
302
quantcore/universe/sources.py
Normal file
@ -0,0 +1,302 @@
|
|||||||
|
"""Candidate lists for the investment universe.
|
||||||
|
|
||||||
|
Two curated index memberships serve as the *candidates* for "the largest 500 US and European
|
||||||
|
stocks":
|
||||||
|
|
||||||
|
* **US**: the S&P 500 constituents (Wikipedia, ``List of S&P 500 companies``).
|
||||||
|
* **Europe** (EU, CH, NO, UK): the STOXX Europe 600 constituents, read from the daily holdings
|
||||||
|
file of the iShares STOXX Europe 600 UCITS ETF (complete list with country, exchange,
|
||||||
|
currency, GICS sector and index weight). Wikipedia's STOXX 600 page is a partial fallback.
|
||||||
|
|
||||||
|
Every fetch is snapshotted to ``quantcore/data/static/`` so that the setup can be reproduced
|
||||||
|
offline (``load_static_candidates``). All frames share the same columns, see
|
||||||
|
:data:`CANDIDATE_COLUMNS`.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import logging
|
||||||
|
from datetime import date
|
||||||
|
from importlib import resources
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
import requests
|
||||||
|
|
||||||
|
from quantcore.universe.symbols import to_yahoo_symbol
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
CANDIDATE_COLUMNS = [
|
||||||
|
"symbol", # Yahoo symbol
|
||||||
|
"source_ticker", # ticker as written by the source
|
||||||
|
"name",
|
||||||
|
"region", # 'US' | 'Europe'
|
||||||
|
"country", # domicile as given by the source (Yahoo's view is added later)
|
||||||
|
"exchange", # exchange name as given by the source
|
||||||
|
"listing_currency", # currency of the listing as given by the source (may be empty)
|
||||||
|
"gics_sector", # GICS sector, English
|
||||||
|
"gics_sub_industry", # only for the S&P 500
|
||||||
|
"index_weight", # index / ETF weight in %, if the source gives one
|
||||||
|
"source",
|
||||||
|
"source_date",
|
||||||
|
]
|
||||||
|
|
||||||
|
USER_AGENT = "Mozilla/5.0 (compatible; quantcore-setup/0.1; +https://fhgr.ch)"
|
||||||
|
|
||||||
|
SP500_URL = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
|
||||||
|
STOXX600_WIKI_URL = "https://en.wikipedia.org/wiki/STOXX_Europe_600"
|
||||||
|
# iShares STOXX Europe 600 UCITS ETF (DE), holdings download of the German retail site
|
||||||
|
ISHARES_STOXX600_URL = (
|
||||||
|
"https://www.ishares.com/de/privatanleger/de/produkte/251931/"
|
||||||
|
"ishares-stoxx-europe-600-ucits-etf-de-fund/1478358465952.ajax"
|
||||||
|
"?fileType=csv&fileName=EXSA_holdings&dataType=fund"
|
||||||
|
)
|
||||||
|
|
||||||
|
# German labels of the iShares file -> English
|
||||||
|
ISHARES_SECTOR_EN = {
|
||||||
|
"IT": "Information Technology",
|
||||||
|
"Finanzwesen": "Financials",
|
||||||
|
"Gesundheitsversorgung": "Health Care",
|
||||||
|
"Energie": "Energy",
|
||||||
|
"Nichtzyklische Konsumgüter": "Consumer Staples",
|
||||||
|
"Zyklische Konsumgüter": "Consumer Discretionary",
|
||||||
|
"Industrie": "Industrials",
|
||||||
|
"Materialien": "Materials",
|
||||||
|
"Grundstoffe": "Materials",
|
||||||
|
"Kommunikation": "Communication Services",
|
||||||
|
"Versorger": "Utilities",
|
||||||
|
"Immobilien": "Real Estate",
|
||||||
|
}
|
||||||
|
ISHARES_COUNTRY_EN = {
|
||||||
|
"Vereinigtes Königreich": "United Kingdom",
|
||||||
|
"Frankreich": "France",
|
||||||
|
"Deutschland": "Germany",
|
||||||
|
"Schweiz": "Switzerland",
|
||||||
|
"Schweden": "Sweden",
|
||||||
|
"Italien": "Italy",
|
||||||
|
"Niederlande": "Netherlands",
|
||||||
|
"Spanien": "Spain",
|
||||||
|
"Norwegen": "Norway",
|
||||||
|
"Dänemark": "Denmark",
|
||||||
|
"Finnland": "Finland",
|
||||||
|
"Polen": "Poland",
|
||||||
|
"Belgien": "Belgium",
|
||||||
|
"Österreich": "Austria",
|
||||||
|
"Griechenland": "Greece",
|
||||||
|
"Irland": "Ireland",
|
||||||
|
"Portugal": "Portugal",
|
||||||
|
"Luxemburg": "Luxembourg",
|
||||||
|
"Tschechische Republik": "Czech Republic",
|
||||||
|
"Ungarn": "Hungary",
|
||||||
|
"Vereinigte Staaten": "United States",
|
||||||
|
"Bermuda": "Bermuda",
|
||||||
|
"Israel": "Israel",
|
||||||
|
}
|
||||||
|
|
||||||
|
# The countries that make up "Europe" for this universe: EU members + CH + NO + UK.
|
||||||
|
# Domiciles outside (Bermuda, Israel, ...) are excluded even if they are STOXX 600 members.
|
||||||
|
EUROPE_COUNTRIES = {
|
||||||
|
"Austria", "Belgium", "Bulgaria", "Croatia", "Cyprus", "Czech Republic", "Denmark", "Estonia",
|
||||||
|
"Finland", "France", "Germany", "Greece", "Hungary", "Ireland", "Italy", "Latvia", "Lithuania",
|
||||||
|
"Luxembourg", "Malta", "Netherlands", "Poland", "Portugal", "Romania", "Slovakia", "Slovenia",
|
||||||
|
"Spain", "Sweden", "Switzerland", "Norway", "United Kingdom",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
_GERMAN_MONTHS = {"jan": 1, "feb": 2, "mär": 3, "mar": 3, "apr": 4, "mai": 5, "jun": 6, "jul": 7, "aug": 8,
|
||||||
|
"sep": 9, "okt": 10, "nov": 11, "dez": 12}
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_german_date(text: str) -> str | None:
|
||||||
|
"""``'18.Sept.2026'`` -> ``'2026-09-18'`` (``None`` when unparseable)."""
|
||||||
|
parts = [p for p in text.replace(" ", "").split(".") if p]
|
||||||
|
if len(parts) != 3:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
day, year = int(parts[0]), int(parts[2])
|
||||||
|
month = _GERMAN_MONTHS.get(parts[1][:3].lower())
|
||||||
|
return date(year, month, day).isoformat() if month else None
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _get(url: str, timeout: float = 30.0) -> bytes:
|
||||||
|
resp = requests.get(url, headers={"User-Agent": USER_AGENT}, timeout=timeout)
|
||||||
|
resp.raise_for_status()
|
||||||
|
return resp.content
|
||||||
|
|
||||||
|
|
||||||
|
def _empty() -> pd.DataFrame:
|
||||||
|
return pd.DataFrame(columns=CANDIDATE_COLUMNS)
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------------------------------
|
||||||
|
# US: S&P 500 from Wikipedia
|
||||||
|
# ----------------------------------------------------------------------------------------
|
||||||
|
def fetch_sp500_wikipedia(timeout: float = 30.0) -> pd.DataFrame:
|
||||||
|
"""S&P 500 constituents with GICS sector and sub-industry (Wikipedia)."""
|
||||||
|
html = _get(SP500_URL, timeout)
|
||||||
|
tables = pd.read_html(io.BytesIO(html))
|
||||||
|
table = next(t for t in tables if "Symbol" in t.columns and "GICS Sector" in t.columns)
|
||||||
|
today = date.today().isoformat()
|
||||||
|
df = pd.DataFrame(
|
||||||
|
{
|
||||||
|
"symbol": [to_yahoo_symbol(s, country="United States") for s in table["Symbol"].astype(str)],
|
||||||
|
"source_ticker": table["Symbol"].astype(str),
|
||||||
|
"name": table["Security"].astype(str),
|
||||||
|
"region": "US",
|
||||||
|
"country": "United States",
|
||||||
|
"exchange": "",
|
||||||
|
"listing_currency": "USD",
|
||||||
|
"gics_sector": table["GICS Sector"].astype(str),
|
||||||
|
"gics_sub_industry": table["GICS Sub-Industry"].astype(str),
|
||||||
|
"index_weight": float("nan"),
|
||||||
|
"source": "wikipedia_sp500",
|
||||||
|
"source_date": today,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
log.info("S&P 500 candidates from Wikipedia: %d", len(df))
|
||||||
|
return df[CANDIDATE_COLUMNS]
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------------------------------
|
||||||
|
# Europe: STOXX Europe 600 from the iShares ETF holdings file
|
||||||
|
# ----------------------------------------------------------------------------------------
|
||||||
|
def parse_ishares_holdings(text: str, source_date: str | None = None) -> pd.DataFrame:
|
||||||
|
"""Parse the German iShares holdings CSV (``Fondsposition per ...`` header, ``;``-free)."""
|
||||||
|
lines = text.lstrip("").splitlines()
|
||||||
|
if source_date is None:
|
||||||
|
first = lines[0] if lines else ""
|
||||||
|
raw_date = first.split(",", 1)[1].strip().strip('"') if "," in first else ""
|
||||||
|
source_date = _parse_german_date(raw_date) or date.today().isoformat()
|
||||||
|
try:
|
||||||
|
header_idx = next(i for i, line in enumerate(lines) if line.startswith("Emittententicker"))
|
||||||
|
except StopIteration as exc:
|
||||||
|
raise ValueError("iShares holdings file: header line not found") from exc
|
||||||
|
body = "\n".join(lines[header_idx:])
|
||||||
|
raw = pd.read_csv(io.StringIO(body), sep=",", quotechar='"', thousands=".", decimal=",")
|
||||||
|
eq = raw[raw["Anlageklasse"] == "Aktien"].copy()
|
||||||
|
eq = eq[eq["Emittententicker"].notna() & (eq["Emittententicker"].astype(str).str.strip() != "-")]
|
||||||
|
|
||||||
|
def _symbol(row: pd.Series) -> str:
|
||||||
|
try:
|
||||||
|
return to_yahoo_symbol(str(row["Emittententicker"]), exchange=str(row["Börse"]))
|
||||||
|
except ValueError:
|
||||||
|
country = ISHARES_COUNTRY_EN.get(str(row["Standort"]), str(row["Standort"]))
|
||||||
|
return to_yahoo_symbol(str(row["Emittententicker"]), country=country)
|
||||||
|
|
||||||
|
clean = lambda v: str(v).replace("\xa0", " ").strip() # noqa: E731
|
||||||
|
df = pd.DataFrame(
|
||||||
|
{
|
||||||
|
"symbol": eq.apply(_symbol, axis=1),
|
||||||
|
"source_ticker": eq["Emittententicker"].astype(str).map(clean),
|
||||||
|
"name": eq["Name"].astype(str).map(clean).str.title(),
|
||||||
|
"region": "Europe",
|
||||||
|
"country": eq["Standort"].map(lambda c: ISHARES_COUNTRY_EN.get(clean(c), clean(c))),
|
||||||
|
"exchange": eq["Börse"].astype(str).map(clean),
|
||||||
|
"listing_currency": eq["Marktwährung"].astype(str).map(clean),
|
||||||
|
"gics_sector": eq["Sektor"].map(lambda s: ISHARES_SECTOR_EN.get(clean(s), clean(s))),
|
||||||
|
"gics_sub_industry": "",
|
||||||
|
"index_weight": pd.to_numeric(eq["Gewichtung (%)"], errors="coerce"),
|
||||||
|
"source": "ishares_stoxx600",
|
||||||
|
"source_date": source_date,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
df = df.drop_duplicates(subset="symbol").reset_index(drop=True)
|
||||||
|
return df[CANDIDATE_COLUMNS]
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_stoxx600_ishares(timeout: float = 30.0) -> pd.DataFrame:
|
||||||
|
"""STOXX Europe 600 constituents from the iShares ETF holdings (complete, with weights)."""
|
||||||
|
raw = _get(ISHARES_STOXX600_URL, timeout).decode("utf-8-sig", errors="replace")
|
||||||
|
df = parse_ishares_holdings(raw)
|
||||||
|
log.info("STOXX 600 candidates from iShares: %d", len(df))
|
||||||
|
return df
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_stoxx600_wikipedia(timeout: float = 30.0) -> pd.DataFrame:
|
||||||
|
"""Partial STOXX 600 list from Wikipedia (fallback; no exchange, so the suffix is by country)."""
|
||||||
|
html = _get(STOXX600_WIKI_URL, timeout)
|
||||||
|
tables = pd.read_html(io.BytesIO(html))
|
||||||
|
table = next(t for t in tables if "Ticker" in t.columns and "Country" in t.columns)
|
||||||
|
today = date.today().isoformat()
|
||||||
|
rows = []
|
||||||
|
for _, r in table.iterrows():
|
||||||
|
try:
|
||||||
|
symbol = to_yahoo_symbol(str(r["Ticker"]), country=str(r["Country"]))
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
rows.append(
|
||||||
|
{
|
||||||
|
"symbol": symbol,
|
||||||
|
"source_ticker": str(r["Ticker"]),
|
||||||
|
"name": str(r["Company"]),
|
||||||
|
"region": "Europe",
|
||||||
|
"country": str(r["Country"]),
|
||||||
|
"exchange": "",
|
||||||
|
"listing_currency": "",
|
||||||
|
"gics_sector": "",
|
||||||
|
"gics_sub_industry": str(r.get("ICB Sector", "")),
|
||||||
|
"index_weight": float("nan"),
|
||||||
|
"source": "wikipedia_stoxx600",
|
||||||
|
"source_date": today,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
df = pd.DataFrame(rows, columns=CANDIDATE_COLUMNS)
|
||||||
|
log.info("STOXX 600 candidates from Wikipedia: %d", len(df))
|
||||||
|
return df
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------------------------------
|
||||||
|
# Static snapshots shipped with the package
|
||||||
|
# ----------------------------------------------------------------------------------------
|
||||||
|
def _static_path(region: str) -> Path:
|
||||||
|
name = "candidates_us.csv" if region.upper() == "US" else "candidates_europe.csv"
|
||||||
|
return Path(str(resources.files("quantcore.data").joinpath(f"static/{name}")))
|
||||||
|
|
||||||
|
|
||||||
|
def load_static_candidates(region: str) -> pd.DataFrame:
|
||||||
|
"""Candidates snapshotted into the package (offline fallback)."""
|
||||||
|
path = _static_path(region)
|
||||||
|
if not path.exists():
|
||||||
|
log.warning("No static candidate file for region %s (%s)", region, path)
|
||||||
|
return _empty()
|
||||||
|
df = pd.read_csv(path, dtype={"symbol": str, "source_ticker": str})
|
||||||
|
log.info("Static %s candidates: %d (%s)", region, len(df), path.name)
|
||||||
|
return df[CANDIDATE_COLUMNS]
|
||||||
|
|
||||||
|
|
||||||
|
def save_static_candidates(df: pd.DataFrame, region: str, path: Path | None = None) -> Path:
|
||||||
|
"""Refresh the shipped snapshot (called by the setup after a successful web fetch)."""
|
||||||
|
path = path or _static_path(region)
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
df[CANDIDATE_COLUMNS].to_csv(path, index=False)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_candidates(region: str, allow_web: bool = True, timeout: float = 30.0) -> pd.DataFrame:
|
||||||
|
"""Best-effort candidate list: web sources first, then the static snapshot.
|
||||||
|
|
||||||
|
Every web source is wrapped: a failing source is logged and the next one is tried
|
||||||
|
(timeout, retry and fallback -- the resilience the platform requires on external calls).
|
||||||
|
"""
|
||||||
|
region = "US" if region.upper() == "US" else "Europe"
|
||||||
|
fetchers = [fetch_sp500_wikipedia] if region == "US" else [fetch_stoxx600_ishares, fetch_stoxx600_wikipedia]
|
||||||
|
if allow_web:
|
||||||
|
for fetch in fetchers:
|
||||||
|
try:
|
||||||
|
df = fetch(timeout)
|
||||||
|
except Exception as exc: # noqa: BLE001 - any failure -> next source
|
||||||
|
log.warning("Candidate source %s failed: %s", fetch.__name__, exc)
|
||||||
|
continue
|
||||||
|
if len(df) >= 100:
|
||||||
|
if region == "Europe":
|
||||||
|
df = df[df["country"].isin(EUROPE_COUNTRIES)].reset_index(drop=True)
|
||||||
|
try:
|
||||||
|
save_static_candidates(df, region)
|
||||||
|
except OSError as exc: # read-only install
|
||||||
|
log.debug("Could not refresh static snapshot: %s", exc)
|
||||||
|
return df
|
||||||
|
return load_static_candidates(region)
|
||||||
129
quantcore/universe/symbols.py
Normal file
129
quantcore/universe/symbols.py
Normal file
@ -0,0 +1,129 @@
|
|||||||
|
"""Mapping of exchange-local ticker codes to Yahoo Finance symbols.
|
||||||
|
|
||||||
|
Yahoo identifies a listing by ``<code><suffix>`` where the suffix encodes the exchange
|
||||||
|
(``.SW`` SIX, ``.L`` London, ``.DE`` Xetra, ...). Share classes are written with a dash
|
||||||
|
(``NOVO-B.CO``, ``INVE-B.ST``, ``BRK-B``), and London codes lose their trailing dot
|
||||||
|
(``RR.`` -> ``RR.L``, ``BT.A`` -> ``BT-A.L``).
|
||||||
|
|
||||||
|
Known exceptions that no rule catches live in ``data/static/symbol_overrides.json`` and are
|
||||||
|
applied last.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
from functools import lru_cache
|
||||||
|
from importlib import resources
|
||||||
|
|
||||||
|
# iShares exchange names (STOXX 600 holdings file) -> Yahoo suffix
|
||||||
|
EXCHANGE_SUFFIX: dict[str, str] = {
|
||||||
|
"London Stock Exchange": ".L",
|
||||||
|
"Xetra": ".DE",
|
||||||
|
"Deutsche Börse AG": ".DE",
|
||||||
|
"Deutsche Boerse AG": ".DE",
|
||||||
|
"Nyse Euronext - Euronext Paris": ".PA",
|
||||||
|
"Euronext Paris": ".PA",
|
||||||
|
"SIX Swiss Exchange": ".SW",
|
||||||
|
"Nasdaq Omx Nordic": ".ST", # Stockholm
|
||||||
|
"Nasdaq Stockholm": ".ST",
|
||||||
|
"Borsa Italiana": ".MI",
|
||||||
|
"Euronext Amsterdam": ".AS",
|
||||||
|
"Bolsa De Madrid": ".MC",
|
||||||
|
"Oslo Bors Asa": ".OL",
|
||||||
|
"Omx Nordic Exchange Copenhagen A/S": ".CO",
|
||||||
|
"Nasdaq Omx Helsinki Ltd.": ".HE",
|
||||||
|
"Warsaw Stock Exchange/Equities/Main Market": ".WA",
|
||||||
|
"Nyse Euronext - Euronext Brussels": ".BR",
|
||||||
|
"Wiener Boerse Ag": ".VI",
|
||||||
|
"Athens Exchange S.A. Cash Market": ".AT",
|
||||||
|
"Irish Stock Exchange - All Market": ".IR",
|
||||||
|
"Nyse Euronext - Euronext Lisbon": ".LS",
|
||||||
|
"Prague Stock Exchange": ".PR",
|
||||||
|
"Budapest Stock Exchange": ".BD",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Country of an exchange (used when a source only gives the country of the listing)
|
||||||
|
COUNTRY_SUFFIX: dict[str, str] = {
|
||||||
|
"United Kingdom": ".L",
|
||||||
|
"Germany": ".DE",
|
||||||
|
"France": ".PA",
|
||||||
|
"Switzerland": ".SW",
|
||||||
|
"Sweden": ".ST",
|
||||||
|
"Italy": ".MI",
|
||||||
|
"Netherlands": ".AS",
|
||||||
|
"Spain": ".MC",
|
||||||
|
"Norway": ".OL",
|
||||||
|
"Denmark": ".CO",
|
||||||
|
"Finland": ".HE",
|
||||||
|
"Poland": ".WA",
|
||||||
|
"Belgium": ".BR",
|
||||||
|
"Austria": ".VI",
|
||||||
|
"Greece": ".AT",
|
||||||
|
"Ireland": ".IR",
|
||||||
|
"Portugal": ".LS",
|
||||||
|
"Czech Republic": ".PR",
|
||||||
|
"Hungary": ".BD",
|
||||||
|
"Luxembourg": ".PA",
|
||||||
|
"United States": "",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=1)
|
||||||
|
def symbol_overrides() -> dict[str, str]:
|
||||||
|
"""Hand-maintained corrections ``{rule_based_symbol: actual_yahoo_symbol}``."""
|
||||||
|
try:
|
||||||
|
text = resources.files("quantcore.data").joinpath("static/symbol_overrides.json").read_text()
|
||||||
|
except FileNotFoundError:
|
||||||
|
return {}
|
||||||
|
data = json.loads(text)
|
||||||
|
return {k: v for k, v in data.items() if not k.startswith("_")}
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_code(code: str) -> str:
|
||||||
|
code = code.strip().upper()
|
||||||
|
code = re.sub(r"\s+", "-", code) # "NOVO B" -> "NOVO-B"
|
||||||
|
code = code.rstrip(".") # "RR." -> "RR"
|
||||||
|
code = code.replace(".", "-") # "BT.A" -> "BT-A"
|
||||||
|
return code
|
||||||
|
|
||||||
|
|
||||||
|
def to_yahoo_symbol(source_ticker: str, exchange: str | None = None, country: str | None = None) -> str:
|
||||||
|
"""Build the Yahoo symbol for an exchange-local ticker code.
|
||||||
|
|
||||||
|
>>> to_yahoo_symbol("NOVO B", "Omx Nordic Exchange Copenhagen A/S")
|
||||||
|
'NOVO-B.CO'
|
||||||
|
>>> to_yahoo_symbol("BT.A", "London Stock Exchange")
|
||||||
|
'BT-A.L'
|
||||||
|
>>> to_yahoo_symbol("BRK.B", country="United States")
|
||||||
|
'BRK-B'
|
||||||
|
"""
|
||||||
|
code = _clean_code(source_ticker)
|
||||||
|
suffix = None
|
||||||
|
if exchange is not None:
|
||||||
|
suffix = EXCHANGE_SUFFIX.get(exchange.strip())
|
||||||
|
if suffix is None and country is not None:
|
||||||
|
suffix = COUNTRY_SUFFIX.get(country.strip())
|
||||||
|
if suffix is None:
|
||||||
|
raise ValueError(f"Unknown exchange/country for ticker {source_ticker!r}: {exchange!r}/{country!r}")
|
||||||
|
symbol = f"{code}{suffix}"
|
||||||
|
return symbol_overrides().get(symbol, symbol)
|
||||||
|
|
||||||
|
|
||||||
|
def normalise_share_class_name(name: str) -> str:
|
||||||
|
"""Reduce a company name to a key that is equal for its share classes.
|
||||||
|
|
||||||
|
Used to keep only one listing per company (``Alphabet A`` / ``Alphabet C``,
|
||||||
|
``Investor A`` / ``Investor B``). The heuristic strips legal forms and class markers.
|
||||||
|
"""
|
||||||
|
s = name.upper()
|
||||||
|
s = re.sub(r"[^A-Z0-9 ]", " ", s)
|
||||||
|
tokens = s.split()
|
||||||
|
drop = {
|
||||||
|
"INC", "CORP", "CORPORATION", "PLC", "AG", "SA", "SE", "NV", "N", "V", "AB", "ASA", "AS", "SPA",
|
||||||
|
"OYJ", "LTD", "LIMITED", "CO", "COMPANY", "GROUP", "HOLDING", "HOLDINGS", "THE", "CLASS", "CL",
|
||||||
|
"SER", "SERIES", "SHS", "SHARE", "SHARES", "ORD", "REG", "REGISTERED", "BEARER", "PS", "GS",
|
||||||
|
"A", "B", "C", "N", "R", "NEW", "KGAA", "SDB", "DI", "NPV", "ADR", "OY", "ORDINARY",
|
||||||
|
}
|
||||||
|
kept = [t for t in tokens if t not in drop]
|
||||||
|
return " ".join(kept[:3]) if kept else s.strip()
|
||||||
16
scripts/experiments/README.md
Normal file
16
scripts/experiments/README.md
Normal file
@ -0,0 +1,16 @@
|
|||||||
|
# Experiments behind the scenario-sampling section
|
||||||
|
|
||||||
|
These scripts produced the tables in `docs/optimization_model.pdf`, Section "Scenario sets:
|
||||||
|
sampling error, variance reduction and large sets". They run against the local data store
|
||||||
|
(`quantcore setup` first) and print their tables; they are not part of the package or the tests.
|
||||||
|
|
||||||
|
| script | what it measures |
|
||||||
|
|---|---|
|
||||||
|
| `scenario_noise.py N S SEEDS [cont\|int] [kinds] [beta]` | GBM model: plain MC vs antithetic / moment matching / LHS / Sobol' at equal `S`, continuous (weights) or integer problem, against the exact Gaussian optimum (cutting planes) — Table "Sampling noise" |
|
||||||
|
| `empirical_noise.py N S SEEDS BETA [kinds]` | empirical-innovation model: bootstrap vs moment-matched / balanced / rank-stratified resampling, evaluated on a 500 000-draw set |
|
||||||
|
| `is_noise.py N SIZES SEEDS BETAS` | importance sampling (two-stage tilt, defensive mixture) vs moment-matched Sobol' — Table "Importance sampling" |
|
||||||
|
| `scaling.py N SIZES highs\|cuopt` | solve time versus the number of scenarios (HiGHS locally / cuOpt on the Spark) — Table "Solve times" |
|
||||||
|
| `rowgen.py N SIZES [full]` | tail-row generation versus the full MILP |
|
||||||
|
| `gurobi_ref.py N BETA` | the exact Gaussian MISOCP solved directly with Gurobi (needs `gurobipy` and a licence large enough for `N`) — validation of the cutting-plane reference |
|
||||||
|
|
||||||
|
Example: `python scripts/experiments/scenario_noise.py 30 2048 8 cont mc,antithetic,mc+mm,sobol,sobol+mm 0.95`
|
||||||
136
scripts/experiments/empirical_noise.py
Normal file
136
scripts/experiments/empirical_noise.py
Normal file
@ -0,0 +1,136 @@
|
|||||||
|
"""Sampling noise of the mean-CVaR LP (weights) under the empirical-innovation model with different resampling schemes.
|
||||||
|
|
||||||
|
'truth' = a common evaluation set of 500k plain bootstrap draws (sampling error of CVaR for a fixed portfolio ~0.1%).
|
||||||
|
"""
|
||||||
|
import sys, time
|
||||||
|
import numpy as np, pandas as pd
|
||||||
|
from scipy.stats import qmc
|
||||||
|
|
||||||
|
from quantcore.config import get_settings
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.optimization.service import OptimizationService, OptimizationRequest
|
||||||
|
from quantcore.optimization.problem import MeanCVaRProblem, ProblemSpec, OptimizationConstraints, TransactionCosts
|
||||||
|
from quantcore.optimization.solvers import HighsSolver
|
||||||
|
from quantcore.models.covariance import CovarianceModel, symmetric_sqrt, symmetric_inv_sqrt
|
||||||
|
from quantcore.models.simulation import EmpiricalInnovationSimulator
|
||||||
|
from quantcore.measurement.risk import scenario_cvar
|
||||||
|
|
||||||
|
N_ASSETS = int(sys.argv[1]) if len(sys.argv) > 1 else 30
|
||||||
|
S = int(sys.argv[2]) if len(sys.argv) > 2 else 2048
|
||||||
|
N_SEEDS = int(sys.argv[3]) if len(sys.argv) > 3 else 8
|
||||||
|
BETA = float(sys.argv[4]) if len(sys.argv) > 4 else 0.95
|
||||||
|
W, H = 1_000_000.0, 5
|
||||||
|
LAMBDA = 0.05 * W
|
||||||
|
S_EVAL = 500_000
|
||||||
|
|
||||||
|
settings = get_settings(); md = MarketDataService(settings); svc = OptimizationService(settings, md)
|
||||||
|
req = OptimizationRequest(max_assets=N_ASSETS, solver="highs", cash=W)
|
||||||
|
symbols = svc.candidate_symbols(req)
|
||||||
|
mu_ann, _, _ = svc.expected_returns(req, symbols)
|
||||||
|
r = md.returns(symbols, currency="base", freq="daily").dropna()
|
||||||
|
cov_model = CovarianceModel.from_dict(req.covariance)
|
||||||
|
sim = EmpiricalInnovationSimulator(cov_model, max_history=req.innovation_history, standardize=req.innovation_standardize)
|
||||||
|
t0 = time.time()
|
||||||
|
eps = sim.innovations(r)
|
||||||
|
e = eps.values - eps.values.mean(axis=0)
|
||||||
|
e, info = sim._restandardize(e)
|
||||||
|
T, N = e.shape
|
||||||
|
cov = cov_model.estimate(r).values
|
||||||
|
A = symmetric_sqrt(cov)
|
||||||
|
mu_h = (mu_ann / 252).reindex(symbols).values * H
|
||||||
|
prices = md.latest_prices(symbols, currency="base").reindex(symbols).values
|
||||||
|
sectors = md.sectors(symbols).fillna("Unknown").tolist()
|
||||||
|
print(f"N={N} assets, T={T} innovations ({time.time()-t0:.0f}s), S={S}, h={H}, beta={BETA}, {info}")
|
||||||
|
# market-factor ordering of the innovation dates (projection on the first eigenvector of Sigma_T)
|
||||||
|
w1 = np.linalg.eigh(cov)[1][:, -1]
|
||||||
|
factor = e @ (A @ w1) # innovation of the dominant direction
|
||||||
|
rank_order = np.argsort(factor) # dates sorted from worst to best market innovation
|
||||||
|
|
||||||
|
|
||||||
|
def moment_match(x, target_var):
|
||||||
|
x = x - x.mean(axis=0)
|
||||||
|
c = x.T @ x / len(x)
|
||||||
|
return x @ symmetric_inv_sqrt(c) * np.sqrt(target_var)
|
||||||
|
|
||||||
|
|
||||||
|
def summed_innovations(kind, seed):
|
||||||
|
rng = np.random.default_rng(seed)
|
||||||
|
if kind == "bootstrap": # current implementation
|
||||||
|
idx = rng.integers(0, T, size=(S, H))
|
||||||
|
return e[idx].sum(axis=1)
|
||||||
|
if kind == "bootstrap+mm":
|
||||||
|
idx = rng.integers(0, T, size=(S, H))
|
||||||
|
return moment_match(e[idx].sum(axis=1), H)
|
||||||
|
if kind == "balanced+mm": # every date used equally often
|
||||||
|
reps = int(np.ceil(S * H / T))
|
||||||
|
idx = rng.permutation(np.tile(np.arange(T), reps))[: S * H].reshape(S, H)
|
||||||
|
return moment_match(e[idx].sum(axis=1), H)
|
||||||
|
if kind == "lhs-ranks+mm": # stratified on the market-factor rank, per day
|
||||||
|
u = qmc.LatinHypercube(d=H, scramble=True, seed=seed).random(S)
|
||||||
|
idx = rank_order[np.minimum((u * T).astype(int), T - 1)]
|
||||||
|
return moment_match(e[idx].sum(axis=1), H)
|
||||||
|
if kind == "sobol-ranks+mm":
|
||||||
|
u = qmc.Sobol(d=H, scramble=True, seed=seed).random(S)
|
||||||
|
idx = rank_order[np.minimum((u * T).astype(int), T - 1)]
|
||||||
|
return moment_match(e[idx].sum(axis=1), H)
|
||||||
|
if kind == "windows+mm": # deterministic: all overlapping h-day windows
|
||||||
|
idx = np.arange(T - H + 1)[:, None] + np.arange(H)[None, :]
|
||||||
|
return moment_match(e[idx].sum(axis=1), H)
|
||||||
|
if kind == "windows":
|
||||||
|
idx = np.arange(T - H + 1)[:, None] + np.arange(H)[None, :]
|
||||||
|
return e[idx].sum(axis=1)
|
||||||
|
raise ValueError(kind)
|
||||||
|
|
||||||
|
|
||||||
|
def scenarios(kind, seed):
|
||||||
|
return mu_h + summed_innovations(kind, seed) @ A.T
|
||||||
|
|
||||||
|
|
||||||
|
def solve_lp(scen):
|
||||||
|
spec = ProblemSpec(symbols=symbols, prices=prices, scenarios=scen, sectors=sectors, cash=W, holdings=np.zeros(N),
|
||||||
|
confidence=BETA, cvar_limit=LAMBDA, objective="max_return", constraints=OptimizationConstraints(),
|
||||||
|
costs=TransactionCosts(proportional_buy=0.0, proportional_sell=0.0))
|
||||||
|
prob = MeanCVaRProblem(spec)
|
||||||
|
model = prob.build()
|
||||||
|
model.var_types = ["C"] * len(model.var_types)
|
||||||
|
sol = HighsSolver().solve(model, time_limit=300)
|
||||||
|
n = sol.x[prob.i_n]
|
||||||
|
pnl = scen @ (n * prices)
|
||||||
|
return n, float(pnl.mean()), scenario_cvar(-pnl, BETA)[1]
|
||||||
|
|
||||||
|
|
||||||
|
# common evaluation set (plain bootstrap, independent seed)
|
||||||
|
rng_eval = np.random.default_rng(987654321)
|
||||||
|
eval_scen = mu_h + e[rng_eval.integers(0, T, size=(S_EVAL, H))].sum(axis=1) @ A.T
|
||||||
|
print(f"evaluation set: {S_EVAL} draws, mean of column means {eval_scen.mean():.2e}")
|
||||||
|
|
||||||
|
|
||||||
|
def true_stats(n):
|
||||||
|
pnl = eval_scen @ (n * prices)
|
||||||
|
return float(pnl.mean()), scenario_cvar(-pnl, BETA)[1]
|
||||||
|
|
||||||
|
|
||||||
|
results = {}
|
||||||
|
KINDS = sys.argv[5].split(",") if len(sys.argv) > 5 else ["bootstrap", "bootstrap+mm", "balanced+mm", "lhs-ranks+mm", "sobol-ranks+mm"]
|
||||||
|
for kind in KINDS:
|
||||||
|
rows, t0 = [], time.time()
|
||||||
|
for seed in range(1 if kind.startswith("windows") else N_SEEDS):
|
||||||
|
n, epnl_in, cvar_in = solve_lp(scenarios(kind, seed))
|
||||||
|
epnl_t, cvar_t = true_stats(n)
|
||||||
|
rows.append({"epnl_in": epnl_in, "cvar_in": cvar_in, "epnl_true": epnl_t, "cvar_true": cvar_t, "n_pos": int((n > 1e-6).sum()), "w": n * prices / W})
|
||||||
|
df = pd.DataFrame(rows)
|
||||||
|
ws = np.vstack(df["w"].values)
|
||||||
|
pair = [np.abs(ws[i] - ws[j]).sum() for i in range(len(ws)) for j in range(i + 1, len(ws))]
|
||||||
|
results[kind] = {
|
||||||
|
"E[P&L] in-sample mean": df.epnl_in.mean(), "E[P&L] in-sample sd": df.epnl_in.std(),
|
||||||
|
"E[P&L] true mean": df.epnl_true.mean(), "E[P&L] true sd": df.epnl_true.std(),
|
||||||
|
"true CVaR mean": df.cvar_true.mean(), "true CVaR sd": df.cvar_true.std(),
|
||||||
|
"CVaR excess over limit (%)": 100 * (df.cvar_true.mean() / LAMBDA - 1),
|
||||||
|
"positions": df.n_pos.mean(), "turnover between seeds (%)": 100 * np.mean(pair) if pair else float("nan"), "seconds": time.time() - t0,
|
||||||
|
}
|
||||||
|
print(f"{kind:15s} done in {time.time()-t0:.0f}s")
|
||||||
|
|
||||||
|
out = pd.DataFrame(results).T
|
||||||
|
pd.set_option("display.width", 250); pd.set_option("display.float_format", lambda v: f"{v:10.2f}")
|
||||||
|
print(f"\nEMPIRICAL model, continuous (weights) problem, CVaR limit {LAMBDA:.0f} CHF, beta={BETA}")
|
||||||
|
print(out.T.to_string())
|
||||||
46
scripts/experiments/gurobi_ref.py
Normal file
46
scripts/experiments/gurobi_ref.py
Normal file
@ -0,0 +1,46 @@
|
|||||||
|
"""Exact Gaussian mean-CVaR problem in integer shares as a MISOCP with Gurobi (validation of the cutting planes)."""
|
||||||
|
import sys, time
|
||||||
|
import numpy as np, gurobipy as gp
|
||||||
|
from scipy.stats import norm
|
||||||
|
from quantcore.config import get_settings
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.optimization.service import OptimizationService, OptimizationRequest
|
||||||
|
from quantcore.optimization.problem import OptimizationConstraints
|
||||||
|
from quantcore.models.covariance import CovarianceModel
|
||||||
|
|
||||||
|
N_ASSETS = int(sys.argv[1]) if len(sys.argv) > 1 else 30
|
||||||
|
BETA = float(sys.argv[2]) if len(sys.argv) > 2 else 0.95
|
||||||
|
W, H = 1_000_000.0, 5
|
||||||
|
LAMBDA = 0.05 * W
|
||||||
|
settings = get_settings(); md = MarketDataService(settings); svc = OptimizationService(settings, md)
|
||||||
|
req = OptimizationRequest(max_assets=N_ASSETS, scenario_model="gbm", solver="highs", cash=W)
|
||||||
|
symbols = svc.candidate_symbols(req)
|
||||||
|
mu_ann, _, _ = svc.expected_returns(req, symbols)
|
||||||
|
r = md.returns(symbols, currency="base", freq="daily").dropna()
|
||||||
|
cov_h = CovarianceModel.from_dict(req.covariance).estimate(r).values * H
|
||||||
|
mu_h = (mu_ann / 252).reindex(symbols).values * H
|
||||||
|
prices = md.latest_prices(symbols, currency="base").reindex(symbols).values
|
||||||
|
sectors = md.sectors(symbols).fillna("Unknown").tolist()
|
||||||
|
N = len(symbols); k = norm.pdf(norm.ppf(BETA)) / (1 - BETA)
|
||||||
|
L = np.linalg.cholesky(cov_h)
|
||||||
|
|
||||||
|
for integer in (True, False):
|
||||||
|
t0 = time.time()
|
||||||
|
m = gp.Model(); m.Params.OutputFlag = 0; m.Params.MIPGap = 1e-9; m.Params.MIPGapAbs = 1e-6
|
||||||
|
ub = np.floor(0.10 * W / prices)
|
||||||
|
n = m.addMVar(N, vtype="I" if integer else "C", lb=0, ub=ub)
|
||||||
|
t = m.addVar(lb=0)
|
||||||
|
y = m.addMVar(N, lb=-gp.GRB.INFINITY) # y = L' v -> ||y|| = sqrt(v' Sigma v)
|
||||||
|
v = prices * n
|
||||||
|
m.addConstr(y == L.T @ v)
|
||||||
|
m.addConstr(y @ y <= t * t) # second-order cone
|
||||||
|
m.addConstr(k * t <= LAMBDA + mu_h @ v) # Gaussian CVaR <= Lambda
|
||||||
|
m.addConstr(prices @ n <= W)
|
||||||
|
for sec in sorted(set(sectors)):
|
||||||
|
mask = np.array([s == sec for s in sectors], dtype=float)
|
||||||
|
m.addConstr((prices * mask) @ n <= OptimizationConstraints().max_sector_weight * W)
|
||||||
|
m.setObjective(mu_h @ v, gp.GRB.MAXIMIZE)
|
||||||
|
m.optimize()
|
||||||
|
nn = n.X; vv = prices * nn; sd = np.sqrt(vv @ cov_h @ vv)
|
||||||
|
print(f"{'integer' if integer else 'continuous':10s} N={N} beta={BETA}: status {m.Status}, E[P&L]={mu_h @ vv:.2f} CHF, "
|
||||||
|
f"CVaR={-mu_h @ vv + k * sd:.2f}, positions={(nn > 0.5).sum()}, {time.time()-t0:.1f}s")
|
||||||
173
scripts/experiments/is_noise.py
Normal file
173
scripts/experiments/is_noise.py
Normal file
@ -0,0 +1,173 @@
|
|||||||
|
"""Importance sampling for the mean-CVaR LP (weights): two-stage exponential tilt along the loss direction of a
|
||||||
|
first-pass solution, defensive mixture with the nominal density, self-normalised weights, weighted moment matching,
|
||||||
|
exact mean in the objective. Compared with Sobol+mm at equal S against the exact Gaussian optimum."""
|
||||||
|
import sys, time
|
||||||
|
import numpy as np, pandas as pd
|
||||||
|
import scipy.sparse as sps
|
||||||
|
from scipy.optimize import milp, LinearConstraint, Bounds
|
||||||
|
from scipy.stats import norm, qmc
|
||||||
|
from quantcore.config import get_settings
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.optimization.service import OptimizationService, OptimizationRequest
|
||||||
|
from quantcore.optimization.problem import OptimizationConstraints
|
||||||
|
from quantcore.models.covariance import CovarianceModel, symmetric_inv_sqrt
|
||||||
|
|
||||||
|
N_ASSETS = int(sys.argv[1]); SIZES = [int(x) for x in sys.argv[2].split(",")]; N_SEEDS = int(sys.argv[3])
|
||||||
|
BETAS = [float(x) for x in sys.argv[4].split(",")]
|
||||||
|
W = 1_000_000.0; LAMBDA = 0.05 * W; H = 5; WMAX = 0.10; SMAX = 0.25
|
||||||
|
settings = get_settings(); md = MarketDataService(settings); svc = OptimizationService(settings, md)
|
||||||
|
req = OptimizationRequest(max_assets=N_ASSETS, solver="highs", cash=W)
|
||||||
|
symbols = svc.candidate_symbols(req)
|
||||||
|
mu_ann, _, _ = svc.expected_returns(req, symbols)
|
||||||
|
r = md.returns(symbols, currency="base", freq="daily").dropna()
|
||||||
|
cov_h = CovarianceModel.from_dict(req.covariance).estimate(r).values * H
|
||||||
|
mu_h = (mu_ann / 252).reindex(symbols).values * H
|
||||||
|
prices = md.latest_prices(symbols, currency="base").reindex(symbols).values
|
||||||
|
sectors = md.sectors(symbols).fillna("Unknown").tolist()
|
||||||
|
N = len(symbols)
|
||||||
|
vals, vecs = np.linalg.eigh(cov_h); o = np.argsort(vals)[::-1]; A = vecs[:, o] * np.sqrt(vals[o])
|
||||||
|
sector_rows = [(np.array([s == sec for s in sectors], dtype=float)) for sec in sorted(set(sectors))]
|
||||||
|
|
||||||
|
|
||||||
|
def k_of(beta):
|
||||||
|
return norm.pdf(norm.ppf(beta)) / (1 - beta)
|
||||||
|
|
||||||
|
|
||||||
|
def true_stats(n, beta):
|
||||||
|
v = n * prices; mean = v @ mu_h; sd = np.sqrt(v @ cov_h @ v)
|
||||||
|
return mean, -mean + k_of(beta) * sd, sd
|
||||||
|
|
||||||
|
|
||||||
|
def weighted_mm(z, p):
|
||||||
|
m = p @ z
|
||||||
|
zc = z - m
|
||||||
|
c = (zc * p[:, None]).T @ zc
|
||||||
|
return zc @ symmetric_inv_sqrt(c)
|
||||||
|
|
||||||
|
|
||||||
|
def solve_lp(scen, p, beta):
|
||||||
|
"""max mu_h'(n*P) s.t. RU-CVaR with scenario probabilities p <= Lambda, budget, caps, sectors; n continuous."""
|
||||||
|
S = len(scen)
|
||||||
|
nv = N + 1 + S
|
||||||
|
c = np.zeros(nv); c[:N] = -(prices * mu_h) # exact model mean in the objective
|
||||||
|
G = scen * prices[None, :]
|
||||||
|
rows = [np.repeat(np.arange(S), N), np.arange(S), np.arange(S)]
|
||||||
|
cols = [np.tile(np.arange(N), S), np.full(S, N), N + 1 + np.arange(S)]
|
||||||
|
vals_ = [G.ravel(), np.ones(S), np.ones(S)]
|
||||||
|
lb = [np.zeros(S)]; ub = [np.full(S, np.inf)]
|
||||||
|
ri = S
|
||||||
|
rows.append(np.concatenate([[ri], np.full(S, ri)])); cols.append(np.concatenate([[N], N + 1 + np.arange(S)]))
|
||||||
|
vals_.append(np.concatenate([[1.0], p / (1 - beta)])); lb.append([-np.inf]); ub.append([LAMBDA]); ri += 1
|
||||||
|
rows.append(np.full(N, ri)); cols.append(np.arange(N)); vals_.append(prices); lb.append([-np.inf]); ub.append([W]); ri += 1
|
||||||
|
for mask in sector_rows:
|
||||||
|
rows.append(np.full(N, ri)); cols.append(np.arange(N)); vals_.append(prices * mask); lb.append([-np.inf]); ub.append([SMAX * W]); ri += 1
|
||||||
|
Am = sps.csr_matrix((np.concatenate(vals_), (np.concatenate(rows), np.concatenate(cols))), shape=(ri, nv))
|
||||||
|
vlb = np.zeros(nv); vub = np.full(nv, np.inf); vub[:N] = WMAX * W / prices; vlb[N] = -np.inf
|
||||||
|
res = milp(c, constraints=LinearConstraint(Am, np.concatenate(lb), np.concatenate(ub)), integrality=np.zeros(nv),
|
||||||
|
bounds=Bounds(vlb, vub))
|
||||||
|
n = res.x[:N]; alpha = res.x[N]
|
||||||
|
return n, alpha
|
||||||
|
|
||||||
|
|
||||||
|
def scenarios_sobol(S, seed):
|
||||||
|
z = norm.ppf(qmc.Sobol(d=N, scramble=True, seed=seed).random(S))
|
||||||
|
p = np.full(S, 1.0 / S)
|
||||||
|
return mu_h + weighted_mm(z, p) @ A.T, p
|
||||||
|
|
||||||
|
|
||||||
|
def scenarios_is(S, seed, beta, n1, eps=0.5, shift=None):
|
||||||
|
"""Mixture proposal: first eps*S Sobol points shifted by theta along the loss direction of n1, rest nominal."""
|
||||||
|
v = n1 * prices
|
||||||
|
a = A.T @ v # loss = -v'mu - a'Z
|
||||||
|
d = -a / np.linalg.norm(a) # unit direction of increasing loss in Z-space
|
||||||
|
theta = (shift if shift is not None else norm.ppf(beta)) * d
|
||||||
|
z = norm.ppf(qmc.Sobol(d=N, scramble=True, seed=seed).random(S))
|
||||||
|
m = int(round(eps * S))
|
||||||
|
z[:m] += theta
|
||||||
|
# likelihood ratio f/g with g = (1-eps) f + eps f(.-theta): 1 / ((1-eps) + eps exp(theta'z - |theta|^2/2))
|
||||||
|
lr = 1.0 / ((1 - eps) + eps * np.exp(z @ theta - theta @ theta / 2))
|
||||||
|
p = lr / lr.sum()
|
||||||
|
ess = 1.0 / (p ** 2).sum()
|
||||||
|
return mu_h + weighted_mm(z, p) @ A.T, p, ess
|
||||||
|
|
||||||
|
|
||||||
|
def scenarios_is_multi(S, seed, n1, shifts=(1.6449, 2.3263), eps=(0.25, 0.25)):
|
||||||
|
"""Multi-component defensive mixture: nominal + one shifted block per shift, all along the loss direction of n1."""
|
||||||
|
v = n1 * prices
|
||||||
|
a = A.T @ v
|
||||||
|
d = -a / np.linalg.norm(a)
|
||||||
|
z = norm.ppf(qmc.Sobol(d=N, scramble=True, seed=seed).random(S))
|
||||||
|
thetas = [sh * d for sh in shifts]
|
||||||
|
start = 0
|
||||||
|
for th, e in zip(thetas, eps):
|
||||||
|
m = int(round(e * S)); z[start:start + m] += th; start += m
|
||||||
|
dens = (1 - sum(eps)) * np.ones(S)
|
||||||
|
for th, e in zip(thetas, eps):
|
||||||
|
dens = dens + e * np.exp(z @ th - th @ th / 2)
|
||||||
|
lr = 1.0 / dens
|
||||||
|
p = lr / lr.sum()
|
||||||
|
return mu_h + weighted_mm(z, p) @ A.T, p, 1.0 / (p ** 2).sum()
|
||||||
|
|
||||||
|
|
||||||
|
def exact_reference(beta, tol=1e-7):
|
||||||
|
k = k_of(beta)
|
||||||
|
cuts = []
|
||||||
|
ub = WMAX * W / prices
|
||||||
|
c = -(prices * mu_h)
|
||||||
|
base_rows = [prices] + [prices * m for m in sector_rows]
|
||||||
|
base_ub = [W] + [SMAX * W] * len(sector_rows)
|
||||||
|
for it in range(500):
|
||||||
|
Am = np.vstack(base_rows + [g for g, _ in cuts]); hi = np.array(base_ub + [b for _, b in cuts])
|
||||||
|
res = milp(c, constraints=LinearConstraint(Am, np.full(len(hi), -np.inf), hi), integrality=np.zeros(N), bounds=Bounds(np.zeros(N), ub))
|
||||||
|
x = res.x; mean, cvar, sd = true_stats(x, beta)
|
||||||
|
if cvar <= LAMBDA * (1 + tol):
|
||||||
|
return x, mean
|
||||||
|
v = x * prices
|
||||||
|
grad = -prices * mu_h + k * prices * (cov_h @ v) / sd
|
||||||
|
cuts.append((grad, LAMBDA - cvar + grad @ x))
|
||||||
|
raise RuntimeError
|
||||||
|
|
||||||
|
|
||||||
|
market_w = md.market_weights(symbols).reindex(symbols).fillna(0.0).values
|
||||||
|
market_n = market_w * W / prices # cap-weighted candidate portfolio (continuous shares)
|
||||||
|
KINDS = sys.argv[5].split(",") if len(sys.argv) > 5 else ["sobol+mm", "sobol+mm+IS(eps=.5)", "IS-market(z_b)", "IS-market(2.33)"]
|
||||||
|
pd.set_option("display.width", 250); pd.set_option("display.float_format", lambda v: f"{v:10.2f}")
|
||||||
|
for beta in BETAS:
|
||||||
|
x_ref, mean_ref = exact_reference(beta)
|
||||||
|
print(f"\n===== beta={beta}: exact optimum E[P&L]={mean_ref:.2f} CHF, CVaR limit {LAMBDA:.0f}")
|
||||||
|
for S in SIZES:
|
||||||
|
results = {}
|
||||||
|
for kind in KINDS:
|
||||||
|
rows = []
|
||||||
|
t0 = time.time()
|
||||||
|
for seed in range(N_SEEDS):
|
||||||
|
if kind == "sobol+mm":
|
||||||
|
scen, p = scenarios_sobol(S, seed); ess = S
|
||||||
|
n, alpha = solve_lp(scen, p, beta)
|
||||||
|
elif kind == "IS-market(mix)":
|
||||||
|
scen, p, ess = scenarios_is_multi(S, seed + 1000, market_n)
|
||||||
|
n, alpha = solve_lp(scen, p, beta)
|
||||||
|
elif kind.startswith("IS-market"):
|
||||||
|
shift = norm.ppf(beta) if "z_b" in kind else 2.33
|
||||||
|
scen, p, ess = scenarios_is(S, seed + 1000, beta, market_n, eps=0.5, shift=shift)
|
||||||
|
n, alpha = solve_lp(scen, p, beta)
|
||||||
|
else:
|
||||||
|
eps = 0.5 if ".5" in kind else 0.3
|
||||||
|
scen1, p1 = scenarios_sobol(S, seed) # stage 1: direction from a plain solve
|
||||||
|
n1, _ = solve_lp(scen1, p1, beta)
|
||||||
|
scen, p, ess = scenarios_is(S, seed + 1000, beta, n1, eps=eps)
|
||||||
|
n, alpha = solve_lp(scen, p, beta)
|
||||||
|
mean_t, cvar_t, _ = true_stats(n, beta)
|
||||||
|
L = -(scen * prices) @ n
|
||||||
|
rows.append({"epnl_true": mean_t, "cvar_true": cvar_t, "w": n * prices / W, "ess": ess,
|
||||||
|
"tail_pts": int((L > alpha).sum())})
|
||||||
|
df = pd.DataFrame(rows); ws = np.vstack(df["w"].values)
|
||||||
|
pair = [np.abs(ws[i] - ws[j]).sum() for i in range(len(ws)) for j in range(i + 1, len(ws))]
|
||||||
|
results[kind] = {
|
||||||
|
"true E[P&L] mean": df.epnl_true.mean(), "loss vs optimum (CHF)": mean_ref - df.epnl_true.mean(),
|
||||||
|
"true CVaR excess (%)": 100 * (df.cvar_true.mean() / LAMBDA - 1), "true CVaR sd (% of limit)": 100 * df.cvar_true.std() / LAMBDA,
|
||||||
|
"turnover between seeds (%)": 100 * np.mean(pair), "turnover to optimum (%)": 100 * np.mean(np.abs(ws - x_ref * prices / W).sum(axis=1)),
|
||||||
|
"scenarios beyond VaR": df.tail_pts.mean(), "ESS": df.ess.mean(), "seconds": time.time() - t0,
|
||||||
|
}
|
||||||
|
print(f"--- S={S}")
|
||||||
|
print(pd.DataFrame(results).to_string())
|
||||||
125
scripts/experiments/rank_and_size.py
Normal file
125
scripts/experiments/rank_and_size.py
Normal file
@ -0,0 +1,125 @@
|
|||||||
|
"""Two questions about large candidate sets.
|
||||||
|
|
||||||
|
A. Conditioning: with ``N`` stocks and an estimator whose effective window is ``T_eff < N`` the
|
||||||
|
covariance is (numerically) singular. Does the optimiser exploit the near-null directions --
|
||||||
|
does it build a portfolio that looks riskless in sample and is not?
|
||||||
|
|
||||||
|
B. How many scenarios does a problem with ``N`` assets need? Measured against the exact Gaussian
|
||||||
|
optimum (no scenarios) for several ``N`` and ``S``.
|
||||||
|
|
||||||
|
Usage: python scripts/experiments/rank_and_size.py [A|B] [sizes]
|
||||||
|
"""
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from quantcore.config import get_settings
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.measurement.estimators import effective_observations
|
||||||
|
from quantcore.models.covariance import CovarianceModel
|
||||||
|
from quantcore.optimization.service import OptimizationRequest, OptimizationService
|
||||||
|
|
||||||
|
PART = sys.argv[1] if len(sys.argv) > 1 else "A"
|
||||||
|
SIZES = [int(x) for x in sys.argv[2].split(",")] if len(sys.argv) > 2 else [30, 100, 300, 745]
|
||||||
|
W, H, BETA = 1_000_000.0, 5, 0.95
|
||||||
|
|
||||||
|
settings = get_settings()
|
||||||
|
md = MarketDataService(settings)
|
||||||
|
svc = OptimizationService(settings, md)
|
||||||
|
caps = md.market_caps().sort_values(ascending=False)
|
||||||
|
pd.set_option("display.width", 240)
|
||||||
|
|
||||||
|
|
||||||
|
def universe(n):
|
||||||
|
return caps.index[:n].tolist()
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================================
|
||||||
|
if PART == "A":
|
||||||
|
rows = []
|
||||||
|
for n in SIZES:
|
||||||
|
symbols = universe(n)
|
||||||
|
r = md.returns(symbols, currency="base", freq="daily").dropna()
|
||||||
|
for label, model in (("ewm l=0.99", CovarianceModel("ewm", lam=0.99, shrinkage=0.0)),
|
||||||
|
("ewm l=0.99 + auto", CovarianceModel("ewm", lam=0.99, shrinkage="auto")),
|
||||||
|
("rolling 200", CovarianceModel("rolling", window=200, shrinkage=0.0)),
|
||||||
|
("rolling 200 + auto", CovarianceModel("rolling", window=200, shrinkage="auto"))):
|
||||||
|
t0 = time.time()
|
||||||
|
cov = model.estimate(r).values
|
||||||
|
vals = np.linalg.eigvalsh(cov)
|
||||||
|
t_eff = effective_observations(model.method, model.window, model.lam)
|
||||||
|
scale = np.diag(cov).mean()
|
||||||
|
rows.append({
|
||||||
|
"N": n, "estimator": label, "T_eff": round(t_eff),
|
||||||
|
"shrinkage": round(model.intensity(r), 3),
|
||||||
|
"min eig / mean var": vals.min() / scale, "cond": vals.max() / max(vals.min(), 1e-300),
|
||||||
|
"eigs < 1e-6 of max": int((vals < 1e-6 * vals.max()).sum()),
|
||||||
|
"s": round(time.time() - t0, 1),
|
||||||
|
})
|
||||||
|
print("A1. conditioning of the covariance estimate\n")
|
||||||
|
print(pd.DataFrame(rows).to_string(index=False, float_format=lambda v: f"{v:.3g}"))
|
||||||
|
|
||||||
|
# does a singular estimate let the optimiser build a 'riskless' portfolio?
|
||||||
|
print("\nA2. minimum-variance portfolio under the caps: in-sample vs out-of-sample volatility")
|
||||||
|
out = []
|
||||||
|
for n in SIZES:
|
||||||
|
symbols = universe(n)
|
||||||
|
r = md.returns(symbols, currency="base", freq="daily").dropna()
|
||||||
|
truth = CovarianceModel("rolling", window=len(r), shrinkage=0.0).estimate(r).values # all history
|
||||||
|
for label, model in (("ewm l=0.99", CovarianceModel("ewm", lam=0.99, shrinkage=0.0)),
|
||||||
|
("ewm l=0.99 + auto", CovarianceModel("ewm", lam=0.99, shrinkage="auto"))):
|
||||||
|
cov = model.estimate(r).values
|
||||||
|
# long-only, fully invested, at most 10 % per name: minimise w'Cw by projected gradient
|
||||||
|
w = np.full(n, 1.0 / n)
|
||||||
|
cap = max(0.10, 1.5 / n)
|
||||||
|
for _ in range(4000):
|
||||||
|
g = cov @ w
|
||||||
|
w = w - g / (np.abs(g).max() + 1e-18) * 0.01
|
||||||
|
w = np.clip(w, 0, cap)
|
||||||
|
if w.sum() <= 0:
|
||||||
|
w = np.full(n, 1.0 / n)
|
||||||
|
w = w / w.sum()
|
||||||
|
vol_in = np.sqrt(w @ cov @ w * 252) * 100
|
||||||
|
vol_out = np.sqrt(w @ truth @ w * 252) * 100
|
||||||
|
out.append({"N": n, "estimator": label, "in-sample vol (%)": vol_in,
|
||||||
|
"long-history vol (%)": vol_out, "understated by": f"{vol_out / vol_in:.2f}x",
|
||||||
|
"positions > 0.1 %": int((w > 0.001).sum())})
|
||||||
|
print(pd.DataFrame(out).to_string(index=False, float_format=lambda v: f"{v:.2f}"))
|
||||||
|
|
||||||
|
# =============================================================================================
|
||||||
|
if PART == "B":
|
||||||
|
scen_sizes = [int(x) for x in sys.argv[3].split(",")] if len(sys.argv) > 3 else [512, 2048, 8192]
|
||||||
|
seeds = int(sys.argv[4]) if len(sys.argv) > 4 else 6
|
||||||
|
solver = sys.argv[5] if len(sys.argv) > 5 else "highs"
|
||||||
|
rows = []
|
||||||
|
for n in SIZES:
|
||||||
|
symbols = universe(n)
|
||||||
|
base = dict(symbols=symbols, cash=W, horizon="week", cvar_limit=0.05 * W, scenario_model="gbm",
|
||||||
|
solver=solver, confidence=BETA, costs={"proportional_buy": 0.0, "proportional_sell": 0.0})
|
||||||
|
t0 = time.time()
|
||||||
|
exact = svc.optimize(OptimizationRequest(**{**base, "scenario_model": "gaussian_exact", "solver": "highs"}))
|
||||||
|
t_exact = time.time() - t0
|
||||||
|
ref_w = {p["symbol"]: p["weight"] for p in exact.positions}
|
||||||
|
for S in scen_sizes:
|
||||||
|
ws, pnls = [], []
|
||||||
|
t1 = time.time()
|
||||||
|
for seed in range(seeds):
|
||||||
|
res = svc.optimize(OptimizationRequest(**{**base, "n_scenarios": S, "seed": seed}))
|
||||||
|
if res.status != "Optimal":
|
||||||
|
continue
|
||||||
|
ws.append({p["symbol"]: p["weight"] for p in res.positions})
|
||||||
|
pnls.append(res.expected_pnl)
|
||||||
|
def turn(a, b):
|
||||||
|
keys = set(a) | set(b)
|
||||||
|
return sum(abs(a.get(k, 0) - b.get(k, 0)) for k in keys)
|
||||||
|
pair = [turn(ws[i], ws[j]) for i in range(len(ws)) for j in range(i + 1, len(ws))]
|
||||||
|
rows.append({"N": n, "S": S, "S/N": S / n,
|
||||||
|
"turnover between seeds (%)": 100 * np.mean(pair),
|
||||||
|
"turnover to exact (%)": 100 * np.mean([turn(w, ref_w) for w in ws]),
|
||||||
|
"E[P&L] mean": np.mean(pnls), "E[P&L] sd": np.std(pnls, ddof=1),
|
||||||
|
"exact E[P&L]": exact.expected_pnl, "s per solve": (time.time() - t1) / max(len(ws), 1)})
|
||||||
|
print(pd.DataFrame(rows[-1:]).to_string(index=False, float_format=lambda v: f"{v:.2f}"), flush=True)
|
||||||
|
print("\nB. scenarios needed as the candidate set grows (exact Gaussian reference)\n")
|
||||||
|
print(pd.DataFrame(rows).to_string(index=False, float_format=lambda v: f"{v:.2f}"))
|
||||||
115
scripts/experiments/rowgen.py
Normal file
115
scripts/experiments/rowgen.py
Normal file
@ -0,0 +1,115 @@
|
|||||||
|
"""Tail-row generation for the RU mean-CVaR MILP: solve on a working set of scenarios, add violated ones, repeat.
|
||||||
|
|
||||||
|
Full problem (new mode, proportional buy costs c): max (g_full - c p)'n s.t.
|
||||||
|
g_s'n - c p'n + alpha + u_s >= 0 (all s) [u_s >= L_s - alpha, L_s = -(g_s - c p)'n]
|
||||||
|
alpha + sum_s u_s / ((1-beta) S) <= Lambda
|
||||||
|
(1 + c) p'n <= W, p_i n_i <= w_max W, sector caps, n integer >= 0
|
||||||
|
Restricted problem: rows only for s in the working set (u_s = 0 elsewhere) -> a relaxation; optimal for the
|
||||||
|
full problem as soon as no omitted scenario has L_s > alpha*.
|
||||||
|
"""
|
||||||
|
import sys, time
|
||||||
|
import numpy as np
|
||||||
|
import scipy.sparse as sps
|
||||||
|
from scipy.optimize import milp, LinearConstraint, Bounds
|
||||||
|
from scipy.stats import norm, qmc
|
||||||
|
from quantcore.config import get_settings
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.optimization.service import OptimizationService, OptimizationRequest
|
||||||
|
from quantcore.optimization.problem import MeanCVaRProblem, ProblemSpec, OptimizationConstraints, TransactionCosts
|
||||||
|
from quantcore.optimization.solvers import HighsSolver
|
||||||
|
from quantcore.models.covariance import CovarianceModel, symmetric_inv_sqrt
|
||||||
|
|
||||||
|
N_ASSETS = int(sys.argv[1]); SIZES = [int(x) for x in sys.argv[2].split(",")]
|
||||||
|
FULL = len(sys.argv) > 3 and sys.argv[3] == "full"
|
||||||
|
BETA = 0.95; W = 1_000_000.0; LAMBDA = 0.05 * W; H = 5; C = 0.001; WMAX = 0.10; SMAX = 0.25; GAP = 1e-4
|
||||||
|
settings = get_settings(); md = MarketDataService(settings); svc = OptimizationService(settings, md)
|
||||||
|
req = OptimizationRequest(max_assets=N_ASSETS, solver="highs", cash=W)
|
||||||
|
symbols = svc.candidate_symbols(req)
|
||||||
|
mu_ann, _, _ = svc.expected_returns(req, symbols)
|
||||||
|
r = md.returns(symbols, currency="base", freq="daily").dropna()
|
||||||
|
cov_h = CovarianceModel.from_dict(req.covariance).estimate(r).values * H
|
||||||
|
mu_h = (mu_ann / 252).reindex(symbols).values * H
|
||||||
|
prices = md.latest_prices(symbols, currency="base").reindex(symbols).values
|
||||||
|
sectors = md.sectors(symbols).fillna("Unknown").tolist()
|
||||||
|
N = len(symbols)
|
||||||
|
vals, vecs = np.linalg.eigh(cov_h); o = np.argsort(vals)[::-1]; A = vecs[:, o] * np.sqrt(vals[o])
|
||||||
|
|
||||||
|
|
||||||
|
def scen_sobol_mm(S, seed=0):
|
||||||
|
z = norm.ppf(qmc.Sobol(d=N, scramble=True, seed=seed).random(S))
|
||||||
|
z = z - z.mean(0); z = z @ symmetric_inv_sqrt(z.T @ z / S)
|
||||||
|
return np.expm1(mu_h + z @ A.T) # simple returns, as in production
|
||||||
|
|
||||||
|
|
||||||
|
def solve_restricted(gain, work, S_full, mean_gain):
|
||||||
|
"""MILP on the working set `work` (indices into gain rows). Returns n, alpha, objective, seconds."""
|
||||||
|
K = len(work)
|
||||||
|
p = prices
|
||||||
|
# variables: n (N), alpha (1), u (K)
|
||||||
|
nv = N + 1 + K
|
||||||
|
c = np.zeros(nv); c[:N] = -(mean_gain - C * p) # minimise negative expected P&L (net of costs)
|
||||||
|
rows, cols, vals_ = [], [], []
|
||||||
|
lb, ub = [], []
|
||||||
|
ri = 0
|
||||||
|
G = gain[work] - C * p[None, :] # net gain per share in the working scenarios
|
||||||
|
# scenario rows: G_s' n + alpha + u_s >= 0
|
||||||
|
rows.extend(np.repeat(np.arange(K), N)); cols.extend(np.tile(np.arange(N), K)); vals_.extend(G.ravel())
|
||||||
|
rows.extend(np.arange(K)); cols.extend([N] * K); vals_.extend([1.0] * K)
|
||||||
|
rows.extend(np.arange(K)); cols.extend(N + 1 + np.arange(K)); vals_.extend([1.0] * K)
|
||||||
|
lb.extend([0.0] * K); ub.extend([np.inf] * K); ri = K
|
||||||
|
# cvar row: alpha + sum u / ((1-beta) S_full) <= Lambda
|
||||||
|
rows.append(ri); cols.append(N); vals_.append(1.0)
|
||||||
|
rows.extend([ri] * K); cols.extend(N + 1 + np.arange(K)); vals_.extend([1.0 / ((1 - BETA) * S_full)] * K)
|
||||||
|
lb.append(-np.inf); ub.append(LAMBDA); ri += 1
|
||||||
|
# budget (1+c) p'n <= W
|
||||||
|
rows.extend([ri] * N); cols.extend(range(N)); vals_.extend((1 + C) * p); lb.append(-np.inf); ub.append(W); ri += 1
|
||||||
|
# sector caps
|
||||||
|
for sec in sorted(set(sectors)):
|
||||||
|
idx = [i for i, s in enumerate(sectors) if s == sec]
|
||||||
|
rows.extend([ri] * len(idx)); cols.extend(idx); vals_.extend(p[idx]); lb.append(-np.inf); ub.append(SMAX * W); ri += 1
|
||||||
|
Am = sps.csr_matrix((vals_, (rows, cols)), shape=(ri, nv))
|
||||||
|
vlb = np.zeros(nv); vub = np.full(nv, np.inf); vub[:N] = np.floor(WMAX * W / p + 1e-9); vlb[N] = -np.inf
|
||||||
|
integrality = np.zeros(nv); integrality[:N] = 1
|
||||||
|
t0 = time.time()
|
||||||
|
res = milp(c, constraints=LinearConstraint(Am, np.array(lb), np.array(ub)), integrality=integrality,
|
||||||
|
bounds=Bounds(vlb, vub), options={"mip_rel_gap": GAP, "time_limit": 1800})
|
||||||
|
n = np.rint(res.x[:N]); alpha = res.x[N]
|
||||||
|
return n, alpha, -res.fun, time.time() - t0
|
||||||
|
|
||||||
|
|
||||||
|
def row_generation(gain, verbose=True):
|
||||||
|
S = len(gain)
|
||||||
|
mean_gain = gain.mean(0)
|
||||||
|
p = prices
|
||||||
|
# initial working set: the worst 2 (1-beta) S scenarios of the equal-weight (cap-respecting) portfolio
|
||||||
|
n0 = np.floor(W / N / p)
|
||||||
|
L0 = -(gain - C * p) @ n0
|
||||||
|
K0 = int(2 * (1 - BETA) * S)
|
||||||
|
work = np.argsort(L0)[-K0:]
|
||||||
|
t_solve = 0.0
|
||||||
|
for it in range(1, 30):
|
||||||
|
n, alpha, obj, dt = solve_restricted(gain, work, S, mean_gain)
|
||||||
|
t_solve += dt
|
||||||
|
L = -(gain - C * p) @ n
|
||||||
|
viol = np.setdiff1d(np.where(L > alpha + 1e-9)[0], work)
|
||||||
|
if verbose:
|
||||||
|
print(f" iter {it}: |work|={len(work)}, MILP {dt:.1f}s, obj {obj:.2f}, alpha {alpha:.1f}, violated {len(viol)}", flush=True)
|
||||||
|
if len(viol) == 0:
|
||||||
|
u = np.maximum(L - alpha, 0)
|
||||||
|
cvar = alpha + u.sum() / ((1 - BETA) * S)
|
||||||
|
return n, obj, cvar, it, t_solve
|
||||||
|
work = np.union1d(work, viol)
|
||||||
|
raise RuntimeError("no convergence")
|
||||||
|
|
||||||
|
|
||||||
|
print(f"N={N}, beta={BETA}, gap={GAP}")
|
||||||
|
for S in SIZES:
|
||||||
|
gain = scen_sobol_mm(S) * prices[None, :]
|
||||||
|
t0 = time.time()
|
||||||
|
n, obj, cvar, iters, t_solve = row_generation(gain)
|
||||||
|
t_rg = time.time() - t0
|
||||||
|
print(f"S={S:>7}: row generation {iters} iterations, {t_rg:.1f}s total ({t_solve:.1f}s in MILPs), E[P&L]={obj:.2f}, CVaR={cvar:.1f}, positions={(n>0).sum()}", flush=True)
|
||||||
|
if FULL:
|
||||||
|
t0 = time.time()
|
||||||
|
n_f, alpha_f, obj_f, dt = solve_restricted(gain, np.arange(S), S, gain.mean(0))
|
||||||
|
print(f"S={S:>7}: full MILP {dt:.1f}s, E[P&L]={obj_f:.2f}, same shares: {np.array_equal(n, n_f)}, max |dn| = {np.abs(n - n_f).max():.0f}", flush=True)
|
||||||
54
scripts/experiments/scaling.py
Normal file
54
scripts/experiments/scaling.py
Normal file
@ -0,0 +1,54 @@
|
|||||||
|
"""Time scaling of the mean-CVaR LP / MILP in the number of scenarios S (HiGHS locally, optionally cuOpt)."""
|
||||||
|
import sys, time, json, zlib
|
||||||
|
import numpy as np
|
||||||
|
from scipy.stats import norm, qmc
|
||||||
|
from quantcore.config import get_settings
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.optimization.service import OptimizationService, OptimizationRequest
|
||||||
|
from quantcore.optimization.problem import MeanCVaRProblem, ProblemSpec, OptimizationConstraints, TransactionCosts
|
||||||
|
from quantcore.optimization.solvers import HighsSolver, CuOptSolver, to_cuopt_payload
|
||||||
|
from quantcore.models.covariance import CovarianceModel, symmetric_inv_sqrt
|
||||||
|
|
||||||
|
N_ASSETS = int(sys.argv[1]); SIZES = [int(x) for x in sys.argv[2].split(",")]; MODE = sys.argv[3] # highs | cuopt
|
||||||
|
BETA = 0.95; W = 1_000_000.0; LAMBDA = 0.05 * W; H = 5
|
||||||
|
settings = get_settings(); md = MarketDataService(settings); svc = OptimizationService(settings, md)
|
||||||
|
req = OptimizationRequest(max_assets=N_ASSETS, solver="highs", cash=W)
|
||||||
|
symbols = svc.candidate_symbols(req)
|
||||||
|
mu_ann, _, _ = svc.expected_returns(req, symbols)
|
||||||
|
r = md.returns(symbols, currency="base", freq="daily").dropna()
|
||||||
|
cov_h = CovarianceModel.from_dict(req.covariance).estimate(r).values * H
|
||||||
|
mu_h = (mu_ann / 252).reindex(symbols).values * H
|
||||||
|
prices = md.latest_prices(symbols, currency="base").reindex(symbols).values
|
||||||
|
sectors = md.sectors(symbols).fillna("Unknown").tolist()
|
||||||
|
N = len(symbols)
|
||||||
|
vals, vecs = np.linalg.eigh(cov_h); o = np.argsort(vals)[::-1]; A = vecs[:, o] * np.sqrt(vals[o])
|
||||||
|
|
||||||
|
def scen_sobol_mm(S, seed=0):
|
||||||
|
z = norm.ppf(qmc.Sobol(d=N, scramble=True, seed=seed).random(S))
|
||||||
|
z = z - z.mean(0); z = z @ symmetric_inv_sqrt(z.T @ z / S)
|
||||||
|
return mu_h + z @ A.T
|
||||||
|
|
||||||
|
print(f"N={N}, beta={BETA}, mode={MODE}")
|
||||||
|
print(f"{'S':>8} {'nnz':>10} {'gen s':>7} {'build s':>8} {'LP s':>8} {'MILP s':>8} {'status':>10} {'E[P&L]':>9} {'json MB':>8} {'cuopt s':>8} {'cu solve':>9}")
|
||||||
|
for S in SIZES:
|
||||||
|
t0 = time.time(); scen = scen_sobol_mm(S); t_gen = time.time() - t0
|
||||||
|
spec = ProblemSpec(symbols=symbols, prices=prices, scenarios=scen, sectors=sectors, cash=W, holdings=np.zeros(N),
|
||||||
|
confidence=BETA, cvar_limit=LAMBDA, objective="max_return", constraints=OptimizationConstraints(),
|
||||||
|
costs=TransactionCosts())
|
||||||
|
prob = MeanCVaRProblem(spec)
|
||||||
|
t0 = time.time(); model = prob.build(); t_build = time.time() - t0
|
||||||
|
nnz = model.A.nnz
|
||||||
|
row = f"{S:>8} {nnz:>10} {t_gen:>7.1f} {t_build:>8.1f}"
|
||||||
|
if MODE == "highs":
|
||||||
|
lp = MeanCVaRProblem(spec).build(); lp.var_types = ["C"] * len(lp.var_types)
|
||||||
|
t0 = time.time(); sol_lp = HighsSolver().solve(lp, time_limit=300); t_lp = time.time() - t0
|
||||||
|
t0 = time.time(); sol = HighsSolver().solve(model, time_limit=300, mip_gap=1e-4); t_mip = time.time() - t0
|
||||||
|
obj = prob.decode(sol.x)["expected_pnl"] if sol.x is not None else float("nan")
|
||||||
|
row += f" {t_lp:>8.1f} {t_mip:>8.1f} {sol.status:>10} {obj:>9.1f}"
|
||||||
|
else:
|
||||||
|
t0 = time.time(); payload = to_cuopt_payload(model, 300.0, {"mip_relative_gap": 1e-4, "log_to_console": False})
|
||||||
|
body = json.dumps(payload); mb = len(body) / 1e6; t_json = time.time() - t0
|
||||||
|
t0 = time.time(); sol = CuOptSolver(settings).solve(model, time_limit=300.0, extra_config={"mip_relative_gap": 1e-4, "log_to_console": False}); t_cu = time.time() - t0
|
||||||
|
obj = prob.decode(sol.x)["expected_pnl"] if sol.x is not None else float("nan")
|
||||||
|
row += f" {'':>8} {'':>8} {sol.status:>10} {obj:>9.1f} {mb:>8.1f} {t_cu:>8.1f} {sol.solve_time:>9.1f}"
|
||||||
|
print(row, flush=True)
|
||||||
177
scripts/experiments/scenario_noise.py
Normal file
177
scripts/experiments/scenario_noise.py
Normal file
@ -0,0 +1,177 @@
|
|||||||
|
"""Sampling noise of the mean-CVaR MILP under different scenario generators (GBM model).
|
||||||
|
|
||||||
|
Reference: the exact Gaussian problem max mu'x s.t. -mu'x + k_beta * sqrt(x'Sigma x) <= Lambda,
|
||||||
|
caps, budget, integer shares -- solved with an outer-approximation (Kelley cutting planes) on the
|
||||||
|
same MILP (no scenarios at all). Under a normal model this is the mean-variance solution.
|
||||||
|
"""
|
||||||
|
import sys, time, json
|
||||||
|
import numpy as np, pandas as pd
|
||||||
|
from scipy.stats import norm, qmc
|
||||||
|
from scipy.optimize import milp, LinearConstraint, Bounds
|
||||||
|
import scipy.sparse as sps
|
||||||
|
|
||||||
|
from quantcore.config import get_settings
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.optimization.service import OptimizationService, OptimizationRequest
|
||||||
|
from quantcore.optimization.problem import MeanCVaRProblem, ProblemSpec, OptimizationConstraints, TransactionCosts
|
||||||
|
from quantcore.optimization.solvers import HighsSolver
|
||||||
|
from quantcore.models.covariance import CovarianceModel, symmetric_inv_sqrt
|
||||||
|
|
||||||
|
N_ASSETS = int(sys.argv[1]) if len(sys.argv) > 1 else 30
|
||||||
|
S = int(sys.argv[2]) if len(sys.argv) > 2 else 2048
|
||||||
|
N_SEEDS = int(sys.argv[3]) if len(sys.argv) > 3 else 8
|
||||||
|
INTEGER = (sys.argv[4] == "int") if len(sys.argv) > 4 else False
|
||||||
|
KINDS = sys.argv[5].split(",") if len(sys.argv) > 5 else ["mc", "antithetic", "mc+mm", "lhs", "sobol", "sobol+mm"]
|
||||||
|
BETA = float(sys.argv[6]) if len(sys.argv) > 6 else 0.95
|
||||||
|
W = 1_000_000.0
|
||||||
|
LAMBDA = 0.05 * W
|
||||||
|
H = 5
|
||||||
|
|
||||||
|
settings = get_settings()
|
||||||
|
md = MarketDataService(settings)
|
||||||
|
svc = OptimizationService(settings, md)
|
||||||
|
req = OptimizationRequest(max_assets=N_ASSETS, scenario_model="gbm", n_scenarios=S, solver="highs", cash=W)
|
||||||
|
symbols = svc.candidate_symbols(req)
|
||||||
|
mu_ann, _, _ = svc.expected_returns(req, symbols)
|
||||||
|
r = md.returns(symbols, currency="base", freq="daily").dropna()
|
||||||
|
cov_model = CovarianceModel.from_dict(req.covariance)
|
||||||
|
cov_d = cov_model.estimate(r).values
|
||||||
|
mu_h = (mu_ann / 252).reindex(symbols).values * H
|
||||||
|
cov_h = cov_d * H
|
||||||
|
prices = md.latest_prices(symbols, currency="base").reindex(symbols).values
|
||||||
|
sectors = md.sectors(symbols).fillna("Unknown").tolist()
|
||||||
|
N = len(symbols)
|
||||||
|
k_beta = norm.pdf(norm.ppf(BETA)) / (1 - BETA)
|
||||||
|
print(f"N={N} assets, S={S} scenarios, horizon {H} days, beta={BETA}, k_beta={k_beta:.4f}")
|
||||||
|
print("weekly vol range (%%): %.2f .. %.2f" % (100 * np.sqrt(np.diag(cov_h)).min(), 100 * np.sqrt(np.diag(cov_h)).max()))
|
||||||
|
|
||||||
|
# eigen root (PCA ordering) and Cholesky root
|
||||||
|
vals, vecs = np.linalg.eigh(cov_h)
|
||||||
|
order = np.argsort(vals)[::-1]
|
||||||
|
A_pca = vecs[:, order] * np.sqrt(vals[order]) # columns = principal directions, largest first
|
||||||
|
A_chol = np.linalg.cholesky(cov_h)
|
||||||
|
|
||||||
|
|
||||||
|
def moment_match(z):
|
||||||
|
z = z - z.mean(axis=0)
|
||||||
|
c = z.T @ z / len(z)
|
||||||
|
return z @ symmetric_inv_sqrt(c)
|
||||||
|
|
||||||
|
|
||||||
|
def base_sample(kind, seed):
|
||||||
|
rng = np.random.default_rng(seed)
|
||||||
|
if kind == "mc":
|
||||||
|
return rng.standard_normal((S, N)), A_chol
|
||||||
|
if kind == "antithetic":
|
||||||
|
z = rng.standard_normal((S // 2, N))
|
||||||
|
return np.vstack([z, -z]), A_chol
|
||||||
|
if kind == "mc+mm":
|
||||||
|
return moment_match(rng.standard_normal((S, N))), A_chol
|
||||||
|
if kind == "lhs":
|
||||||
|
u = qmc.LatinHypercube(d=N, scramble=True, seed=seed).random(S)
|
||||||
|
return norm.ppf(u), A_pca
|
||||||
|
if kind == "sobol":
|
||||||
|
u = qmc.Sobol(d=N, scramble=True, seed=seed).random(S)
|
||||||
|
return norm.ppf(u), A_pca
|
||||||
|
if kind == "sobol+mm":
|
||||||
|
u = qmc.Sobol(d=N, scramble=True, seed=seed).random(S)
|
||||||
|
return moment_match(norm.ppf(u)), A_pca
|
||||||
|
raise ValueError(kind)
|
||||||
|
|
||||||
|
|
||||||
|
def scenarios(kind, seed):
|
||||||
|
z, a = base_sample(kind, seed)
|
||||||
|
return mu_h + z @ a.T # S x N horizon log-returns (used linearly: exact Gaussian P&L)
|
||||||
|
|
||||||
|
|
||||||
|
def solve_milp(scen, integer):
|
||||||
|
spec = ProblemSpec(symbols=symbols, prices=prices, scenarios=scen, sectors=sectors, cash=W,
|
||||||
|
holdings=np.zeros(N), confidence=BETA, cvar_limit=LAMBDA, objective="max_return",
|
||||||
|
constraints=OptimizationConstraints(), costs=TransactionCosts(proportional_buy=0.0, proportional_sell=0.0))
|
||||||
|
prob = MeanCVaRProblem(spec)
|
||||||
|
model = prob.build()
|
||||||
|
if not integer:
|
||||||
|
model.var_types = ["C"] * len(model.var_types)
|
||||||
|
sol = HighsSolver().solve(model, time_limit=300, mip_gap=1e-5)
|
||||||
|
n = np.rint(sol.x[prob.i_n]) if integer else sol.x[prob.i_n]
|
||||||
|
pnl = scen @ (n * prices)
|
||||||
|
from quantcore.measurement.risk import scenario_cvar
|
||||||
|
_, cvar_in = scenario_cvar(-pnl, BETA)
|
||||||
|
return n, float(pnl.mean()), cvar_in, sol
|
||||||
|
|
||||||
|
|
||||||
|
def true_stats(n):
|
||||||
|
v = n * prices
|
||||||
|
mean = v @ mu_h
|
||||||
|
sd = np.sqrt(v @ cov_h @ v)
|
||||||
|
return mean, -mean + k_beta * sd, sd
|
||||||
|
|
||||||
|
|
||||||
|
# ---- exact Gaussian reference by outer approximation on the same integer feasible set ----------------
|
||||||
|
def exact_reference(tol=1e-6, max_iter=200, integer=True):
|
||||||
|
ub = np.floor(np.minimum(0.10, 1.0) * W / prices)
|
||||||
|
c = -(prices * mu_h) # minimise -E[P&L]
|
||||||
|
rows = [prices] # budget
|
||||||
|
lb = [-np.inf]; ubs = [W]
|
||||||
|
for sec in sorted(set(sectors)): # sector caps (defaults of OptimizationConstraints)
|
||||||
|
mask = np.array([s == sec for s in sectors], dtype=float)
|
||||||
|
rows.append(prices * mask); lb.append(-np.inf); ubs.append(OptimizationConstraints().max_sector_weight * W)
|
||||||
|
cuts = []
|
||||||
|
n_prev = None
|
||||||
|
for it in range(max_iter):
|
||||||
|
A = np.vstack(rows + [cut[0] for cut in cuts])
|
||||||
|
lo = np.array(lb + [-np.inf] * len(cuts)); hi = np.array(ubs + [cut[1] for cut in cuts])
|
||||||
|
res = milp(c, constraints=LinearConstraint(A, lo, hi), integrality=np.ones(N) if integer else np.zeros(N),
|
||||||
|
bounds=Bounds(np.zeros(N), ub), options={"mip_rel_gap": 1e-9})
|
||||||
|
n = np.round(res.x) if integer else res.x
|
||||||
|
mean, cvar, sd = true_stats(n)
|
||||||
|
if cvar <= LAMBDA + tol * LAMBDA:
|
||||||
|
return n, mean, cvar, it + 1
|
||||||
|
v = n * prices
|
||||||
|
grad = -prices * mu_h + k_beta * prices * (cov_h @ v) / sd # d CVaR / d n
|
||||||
|
# cut: cvar(n_k) + grad'(n - n_k) <= Lambda -> grad'n <= Lambda - cvar(n_k) + grad'n_k
|
||||||
|
cuts.append((grad, LAMBDA - cvar + grad @ n))
|
||||||
|
raise RuntimeError("no convergence")
|
||||||
|
|
||||||
|
|
||||||
|
t0 = time.time()
|
||||||
|
n_ref, mean_ref, cvar_ref, iters = exact_reference()
|
||||||
|
x_rel, mean_rel, cvar_rel, iters_rel = exact_reference(integer=False)
|
||||||
|
print(f"\nexact Gaussian reference (integer): E[P&L]={mean_ref:.2f} CHF, CVaR={cvar_ref:.2f}, "
|
||||||
|
f"{int((n_ref > 0).sum())} positions, {iters} cuts, {time.time()-t0:.1f}s")
|
||||||
|
print(f"continuous relaxation: E[P&L]={mean_rel:.2f} CHF, CVaR={cvar_rel:.2f}")
|
||||||
|
|
||||||
|
results = {}
|
||||||
|
for kind in KINDS:
|
||||||
|
rows = []
|
||||||
|
t0 = time.time()
|
||||||
|
for seed in range(N_SEEDS):
|
||||||
|
n, epnl, cvar_in, sol = solve_milp(scenarios(kind, seed), INTEGER)
|
||||||
|
mean_t, cvar_t, _ = true_stats(n)
|
||||||
|
rows.append({"seed": seed, "epnl_in": epnl, "cvar_in": cvar_in, "epnl_true": mean_t, "cvar_true": cvar_t,
|
||||||
|
"n_pos": int((n > 0).sum()), "shares": n, "status": sol.status})
|
||||||
|
df = pd.DataFrame(rows)
|
||||||
|
shares = np.vstack(df["shares"].values)
|
||||||
|
# dissimilarity between seeds: mean over pairs of (value traded to go from one to the other) / W
|
||||||
|
pair_turn = []
|
||||||
|
for i in range(len(shares)):
|
||||||
|
for j in range(i + 1, len(shares)):
|
||||||
|
pair_turn.append(np.abs(shares[i] - shares[j]) @ prices / W)
|
||||||
|
dist_ref = np.abs(shares - (n_ref if INTEGER else x_rel)) @ prices / W
|
||||||
|
results[kind] = {
|
||||||
|
"E[P&L] in-sample mean": df.epnl_in.mean(), "E[P&L] in-sample sd": df.epnl_in.std(),
|
||||||
|
"E[P&L] true mean": df.epnl_true.mean(), "E[P&L] true sd": df.epnl_true.std(),
|
||||||
|
"loss vs ref (CHF)": (mean_ref if INTEGER else mean_rel) - df.epnl_true.mean(),
|
||||||
|
"true CVaR mean": df.cvar_true.mean(), "true CVaR sd": df.cvar_true.std(),
|
||||||
|
"CVaR excess over limit (%)": 100 * (df.cvar_true.mean() / LAMBDA - 1),
|
||||||
|
"positions": df.n_pos.mean(),
|
||||||
|
"turnover between seeds (%)": 100 * np.mean(pair_turn), "turnover to ref (%)": 100 * dist_ref.mean(),
|
||||||
|
"seconds": time.time() - t0,
|
||||||
|
}
|
||||||
|
print(f"{kind:10s} done in {time.time()-t0:.0f}s statuses={df.status.unique().tolist()}")
|
||||||
|
|
||||||
|
out = pd.DataFrame(results).T
|
||||||
|
pd.set_option("display.width", 250); pd.set_option("display.float_format", lambda v: f"{v:10.2f}")
|
||||||
|
print("\n%s problem; reference E[P&L] = %.2f CHF, CVaR limit = %.0f CHF (5%% of wealth), weekly" % ("INTEGER" if INTEGER else "CONTINUOUS (weights)", mean_ref if INTEGER else mean_rel, LAMBDA))
|
||||||
|
print(out.T.to_string())
|
||||||
|
out.to_csv(f"{'int' if INTEGER else 'cont'}_scenario_noise_N{N}_S{S}.csv")
|
||||||
155
scripts/experiments/tilt_correlation.py
Normal file
155
scripts/experiments/tilt_correlation.py
Normal file
@ -0,0 +1,155 @@
|
|||||||
|
"""What the importance-sampling tilt does with the correlation structure.
|
||||||
|
|
||||||
|
1. The per-asset shift is proportional to ``Sigma v`` (covariance with the reference portfolio),
|
||||||
|
not to the assets' own volatilities.
|
||||||
|
2. The tilted set keeps the full covariance (an exponential tilt of a Gaussian is a mean shift;
|
||||||
|
moment matching makes it exact), so correlations are untouched.
|
||||||
|
3. How much tail coverage does a portfolio get whose loss direction differs from the reference's?
|
||||||
|
Measured for portfolios ranging from market-like to deliberately orthogonal.
|
||||||
|
|
||||||
|
Usage: python scripts/experiments/tilt_correlation.py [N] [S] [SEEDS] [BETA]
|
||||||
|
"""
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
from scipy.stats import norm
|
||||||
|
|
||||||
|
from quantcore.config import get_settings
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.measurement.risk import scenario_cvar
|
||||||
|
from quantcore.models import sampling as smp
|
||||||
|
from quantcore.models.covariance import CovarianceModel
|
||||||
|
from quantcore.models.simulation import GBMSimulator, TiltSpec
|
||||||
|
from quantcore.optimization.service import OptimizationRequest, OptimizationService
|
||||||
|
|
||||||
|
N_ASSETS = int(sys.argv[1]) if len(sys.argv) > 1 else 30
|
||||||
|
S = int(sys.argv[2]) if len(sys.argv) > 2 else 4096
|
||||||
|
N_SEEDS = int(sys.argv[3]) if len(sys.argv) > 3 else 12
|
||||||
|
BETA = float(sys.argv[4]) if len(sys.argv) > 4 else 0.95
|
||||||
|
W, H = 1_000_000.0, 5
|
||||||
|
|
||||||
|
settings = get_settings()
|
||||||
|
md = MarketDataService(settings)
|
||||||
|
svc = OptimizationService(settings, md)
|
||||||
|
req = OptimizationRequest(max_assets=N_ASSETS, scenario_model="gbm", n_scenarios=S, horizon="week", cvar_limit=0.05 * W)
|
||||||
|
symbols = svc.candidate_symbols(req)
|
||||||
|
mu_ann, _, _ = svc.expected_returns(req, symbols)
|
||||||
|
r = md.returns(symbols, currency="base", freq="daily").dropna()
|
||||||
|
cov_model = CovarianceModel.from_dict(req.covariance)
|
||||||
|
cov_h = cov_model.estimate(r).values * H
|
||||||
|
prices = md.latest_prices(symbols, currency="base").reindex(symbols).values
|
||||||
|
sectors = md.sectors(symbols).fillna("Unknown")
|
||||||
|
N = len(symbols)
|
||||||
|
vol_h = np.sqrt(np.diag(cov_h))
|
||||||
|
market_w = svc.market_exposure(symbols)
|
||||||
|
|
||||||
|
print(f"N={N}, S={S}, beta={BETA}, weekly horizon; average correlation "
|
||||||
|
f"{np.mean((cov_h / np.outer(vol_h, vol_h))[np.triu_indices(N, 1)]):.2f}")
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# 1. what the tilt does per asset
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
A = smp.pca_root(cov_h)
|
||||||
|
v_ref = market_w * W
|
||||||
|
d = smp.loss_direction(A, v_ref)
|
||||||
|
shift_returns = norm.ppf(BETA) * (A @ d) # the return shift of the tilted block
|
||||||
|
sigma_ref = np.sqrt(v_ref @ cov_h @ v_ref)
|
||||||
|
implied = -norm.ppf(BETA) * (cov_h @ v_ref) / sigma_ref # -z_beta * Sigma v / sigma
|
||||||
|
beta_i = (cov_h @ market_w) / (market_w @ cov_h @ market_w) # beta of each asset to the reference
|
||||||
|
table = pd.DataFrame({
|
||||||
|
"weight (%)": market_w * 100, "weekly vol (%)": vol_h * 100, "beta to ref": beta_i,
|
||||||
|
"shift (%)": shift_returns * 100, "shift / own vol": shift_returns / vol_h,
|
||||||
|
}, index=symbols)
|
||||||
|
print("\n1. per-asset shift of the tilted block (z_beta along the loss direction)")
|
||||||
|
print(" max |A d - (-z Sigma v / sigma)| = %.2e (the shift *is* -z_beta Sigma v / sigma)"
|
||||||
|
% np.abs(shift_returns - implied).max())
|
||||||
|
print(" corr(shift, beta) = %.4f, corr(shift, own vol) = %.4f"
|
||||||
|
% (np.corrcoef(shift_returns, beta_i)[0, 1], np.corrcoef(shift_returns, vol_h)[0, 1]))
|
||||||
|
show = table.sort_values("beta to ref")
|
||||||
|
print(pd.concat([show.head(4), show.tail(4)]).round(3).to_string())
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# 2. the tilted set keeps the covariance
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
sim = GBMSimulator(cov_model)
|
||||||
|
plain = sim.simulate(r, n_scenarios=S, horizon_periods=H, mu=mu_ann / 252, seed=1)
|
||||||
|
tilted = sim.simulate(r, n_scenarios=S, horizon_periods=H, mu=mu_ann / 252, seed=1,
|
||||||
|
tilt=TiltSpec(exposure=v_ref, label="market"))
|
||||||
|
w = tilted.weights
|
||||||
|
centred = tilted.log_returns - w @ tilted.log_returns
|
||||||
|
cov_t = (centred * w[:, None]).T @ centred
|
||||||
|
corr_t = cov_t / np.sqrt(np.outer(np.diag(cov_t), np.diag(cov_t)))
|
||||||
|
corr_m = cov_h / np.outer(vol_h, vol_h)
|
||||||
|
# unweighted (i.e. ignoring the likelihood ratios) for comparison
|
||||||
|
cov_u = np.cov(tilted.log_returns, rowvar=False)
|
||||||
|
corr_u = cov_u / np.sqrt(np.outer(np.diag(cov_u), np.diag(cov_u)))
|
||||||
|
print("\n2. covariance of the tilted set (weighted with the likelihood ratios)")
|
||||||
|
print(" max |weighted cov - h Sigma| = %.2e" % np.abs(cov_t - cov_h).max())
|
||||||
|
print(" max |weighted corr - model corr| = %.2e" % np.abs(corr_t - corr_m).max())
|
||||||
|
print(" max |UNweighted corr - model| = %.2e (the raw tilted sample is not the model)"
|
||||||
|
% np.abs(corr_u - corr_m).max())
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# 3. coverage of portfolios that are not market-like
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def portfolio(kind: str) -> np.ndarray:
|
||||||
|
if kind == "market":
|
||||||
|
return market_w.copy()
|
||||||
|
if kind == "equal weight":
|
||||||
|
return np.full(N, 1.0 / N)
|
||||||
|
if kind == "optimised":
|
||||||
|
res = svc.optimize(OptimizationRequest(max_assets=N_ASSETS, scenario_model="gbm", n_scenarios=S,
|
||||||
|
horizon="week", cvar_limit=0.05 * W, cash=W, solver="highs",
|
||||||
|
confidence=BETA))
|
||||||
|
held = {p["symbol"]: p["value"] for p in res.positions}
|
||||||
|
v = np.array([held.get(s, 0.0) for s in symbols])
|
||||||
|
return v / v.sum()
|
||||||
|
if kind == "highest beta":
|
||||||
|
idx = np.argsort(beta_i)[-5:]
|
||||||
|
v = np.zeros(N); v[idx] = 1 / len(idx); return v
|
||||||
|
if kind == "lowest beta":
|
||||||
|
idx = np.argsort(beta_i)[:5]
|
||||||
|
v = np.zeros(N); v[idx] = 1 / len(idx); return v
|
||||||
|
if kind == "min variance":
|
||||||
|
inv = np.linalg.pinv(cov_h) @ np.ones(N)
|
||||||
|
v = np.maximum(inv, 0); return v / v.sum()
|
||||||
|
if kind == "least market-like": # maximise the angle to the reference loss direction, long only
|
||||||
|
c = cov_h @ market_w
|
||||||
|
v = np.zeros(N); v[np.argsort(c / vol_h)[:3]] = 1 / 3; return v
|
||||||
|
raise ValueError(kind)
|
||||||
|
|
||||||
|
|
||||||
|
kinds = ["market", "equal weight", "optimised", "highest beta", "min variance", "lowest beta", "least market-like"]
|
||||||
|
rows = []
|
||||||
|
for kind in kinds:
|
||||||
|
wgt = portfolio(kind)
|
||||||
|
v = wgt * W
|
||||||
|
d_p = smp.loss_direction(A, v)
|
||||||
|
cos = float(d_p @ d) # alignment of the loss directions
|
||||||
|
losses_p, losses_t, cvars_p, cvars_t = [], [], [], []
|
||||||
|
for seed in range(N_SEEDS):
|
||||||
|
pl = sim.simulate(r, n_scenarios=S, horizon_periods=H, mu=mu_ann / 252, seed=seed)
|
||||||
|
ti = sim.simulate(r, n_scenarios=S, horizon_periods=H, mu=mu_ann / 252, seed=seed,
|
||||||
|
tilt=TiltSpec(exposure=v_ref, label="market"))
|
||||||
|
lp = -(pl.simple_returns() @ v)
|
||||||
|
lt = -(ti.simple_returns() @ v)
|
||||||
|
cvars_p.append(scenario_cvar(lp, BETA)[1])
|
||||||
|
cvars_t.append(scenario_cvar(lt, BETA, ti.probabilities)[1])
|
||||||
|
var_p = np.quantile(lp, BETA)
|
||||||
|
losses_p.append((lp > var_p).sum())
|
||||||
|
losses_t.append((lt > var_p).sum())
|
||||||
|
exact = -(v @ (np.expm1(mu_ann.reindex(symbols).values / 252 * H))) + \
|
||||||
|
norm.pdf(norm.ppf(BETA)) / (1 - BETA) * np.sqrt(v @ cov_h @ v)
|
||||||
|
rows.append({
|
||||||
|
"cos to ref direction": cos, "tail points plain": np.mean(losses_p), "tail points tilted": np.mean(losses_t),
|
||||||
|
"CVaR sd plain (%)": 100 * np.std(cvars_p, ddof=1) / np.mean(cvars_p),
|
||||||
|
"CVaR sd tilted (%)": 100 * np.std(cvars_t, ddof=1) / np.mean(cvars_t),
|
||||||
|
"noise reduction": np.std(cvars_p, ddof=1) / max(np.std(cvars_t, ddof=1), 1e-12),
|
||||||
|
"CVaR bias tilted (%)": 100 * (np.mean(cvars_t) / exact - 1),
|
||||||
|
"CVaR bias plain (%)": 100 * (np.mean(cvars_p) / exact - 1),
|
||||||
|
})
|
||||||
|
out = pd.DataFrame(rows, index=kinds)
|
||||||
|
pd.set_option("display.width", 220)
|
||||||
|
print(f"\n3. tail coverage by portfolio type ({N_SEEDS} seeds; 'cos' = alignment of loss directions)")
|
||||||
|
print(out.round(2).to_string())
|
||||||
62
tests/conftest.py
Normal file
62
tests/conftest.py
Normal file
@ -0,0 +1,62 @@
|
|||||||
|
"""Shared fixtures: a synthetic data store with 12 stocks in four currencies."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from quantcore.config import Settings
|
||||||
|
from quantcore.data.market_data import MarketDataService
|
||||||
|
from quantcore.data.store import DataStore
|
||||||
|
|
||||||
|
N, T = 12, 3000
|
||||||
|
SYMBOLS = [f"S{i}" for i in range(N)]
|
||||||
|
CURRENCIES = ["CHF", "USD", "EUR", "GBP"] * 3
|
||||||
|
SECTORS = ["Information Technology", "Financials", "Health Care", "Energy"] * 3
|
||||||
|
|
||||||
|
|
||||||
|
def make_prices(seed: int = 0) -> tuple[pd.DataFrame, np.ndarray]:
|
||||||
|
rng = np.random.default_rng(seed)
|
||||||
|
dates = pd.bdate_range("2013-01-01", periods=T)
|
||||||
|
f = rng.standard_normal((T, 2)) * 0.01
|
||||||
|
beta = rng.uniform(0.5, 1.5, size=(N, 2))
|
||||||
|
eps = rng.standard_t(5, size=(T, N)) * 0.01
|
||||||
|
r = f @ beta.T + eps + 0.0002
|
||||||
|
return pd.DataFrame(100 * np.exp(np.cumsum(r, axis=0)), index=dates, columns=SYMBOLS), r
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="session")
|
||||||
|
def settings(tmp_path_factory) -> Settings:
|
||||||
|
return Settings(data_dir=tmp_path_factory.mktemp("data"), cuopt_url="http://127.0.0.1:9", solver_fallback=True)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="session")
|
||||||
|
def store(settings) -> DataStore:
|
||||||
|
store = DataStore(settings)
|
||||||
|
store.ensure_dirs()
|
||||||
|
prices, _ = make_prices()
|
||||||
|
rng = np.random.default_rng(1)
|
||||||
|
uni = pd.DataFrame({
|
||||||
|
"symbol": SYMBOLS, "name": [f"Company {s}" for s in SYMBOLS], "region": ["US"] * 6 + ["Europe"] * 6,
|
||||||
|
"country": ["United States"] * 6 + ["Switzerland"] * 6, "listing_currency": CURRENCIES, "exchange": "",
|
||||||
|
"exchange_code": "", "gics_sector": SECTORS, "gics_sub_industry": "", "yahoo_sector": "", "yahoo_industry": "",
|
||||||
|
"market_cap": 1e9, "market_cap_base": rng.uniform(1e9, 1e11, N), "shares_outstanding": rng.uniform(1e7, 1e9, N),
|
||||||
|
"last_price": prices.iloc[-1].values, "first_price_date": "2013-01-01", "price_coverage": 1.0,
|
||||||
|
"index_weight": np.nan, "source": "synthetic", "rank_in_region": range(1, N + 1),
|
||||||
|
})
|
||||||
|
store.save_universe(uni)
|
||||||
|
store.save_prices(prices)
|
||||||
|
fx = pd.DataFrame({
|
||||||
|
"USD": 0.85 * np.exp(np.cumsum(rng.standard_normal(T) * 0.004)),
|
||||||
|
"EUR": 0.95 * np.exp(np.cumsum(rng.standard_normal(T) * 0.003)),
|
||||||
|
"GBP": 1.10 * np.exp(np.cumsum(rng.standard_normal(T) * 0.004)), "CHF": 1.0,
|
||||||
|
}, index=prices.index)
|
||||||
|
store.save_fx(fx)
|
||||||
|
store.update_manifest(last_price_date=str(prices.index[-1].date()), n_universe=N)
|
||||||
|
return store
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="session")
|
||||||
|
def md(settings, store) -> MarketDataService:
|
||||||
|
return MarketDataService(settings, store)
|
||||||
209
tests/test_api.py
Normal file
209
tests/test_api.py
Normal file
@ -0,0 +1,209 @@
|
|||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
import quantcore.api.app as app_module
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def client(settings, store):
|
||||||
|
app_module.state = app_module.State(settings)
|
||||||
|
with TestClient(app_module.app) as c:
|
||||||
|
yield c
|
||||||
|
|
||||||
|
|
||||||
|
def test_health_and_universe(client):
|
||||||
|
h = client.get("/health").json()
|
||||||
|
assert h["status"] == "ok" and h["store"]["prices"] is True and h["cuopt"]["reachable"] is False
|
||||||
|
u = client.get("/universe", params={"region": "Europe"}).json()
|
||||||
|
assert len(u) == 6 and all(row["region"] == "Europe" for row in u)
|
||||||
|
assert client.get("/universe/S0").json()["symbol"] == "S0"
|
||||||
|
assert client.get("/universe/NOPE").status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_prices_returns_and_market(client):
|
||||||
|
p = client.get("/prices", params={"symbols": "S0,S1", "freq": "weekly", "start": "2020-01-01"}).json()
|
||||||
|
assert p["columns"] == ["S0", "S1"] and len(p["index"]) > 100
|
||||||
|
r = client.get("/returns", params={"symbols": "S0", "currency": "local"}).json()
|
||||||
|
assert r["kind"] == "log_returns"
|
||||||
|
m = client.get("/market/index", params={"freq": "weekly"}).json()
|
||||||
|
assert "mean_ann" in m and m["n"] > 100
|
||||||
|
w = client.get("/market/weights").json()["weights"]
|
||||||
|
assert abs(sum(w.values()) - 1) < 1e-6
|
||||||
|
|
||||||
|
|
||||||
|
def test_risk_and_performance_endpoints(client):
|
||||||
|
body = {"symbols": ["S0", "S1", "S2"], "horizon": "week", "confidence": [0.95, 0.99]}
|
||||||
|
rows = client.post("/risk/assets", json=body).json()
|
||||||
|
assert len(rows) == 3 and rows[0]["cvar99_hist"] > rows[0]["var99_hist"]
|
||||||
|
corr = client.post("/risk/correlation", json={"symbols": ["S0", "S1"], "method": "ewm", "lam": 0.99}).json()
|
||||||
|
assert corr["data"][0][0] == pytest.approx(1.0)
|
||||||
|
pr = client.post("/risk/portfolio", json={"holdings": {"S0": 10, "S3": 5}, "cash": 500, "horizon": "month"}).json()
|
||||||
|
assert pr["confidence"]["0.95"]["cvar_money"] > 0 and pr["horizon_periods"] == 21
|
||||||
|
series = client.post("/risk/series", json={"symbol": "S0", "window": 100}).json()
|
||||||
|
assert "vol_ewm_ann" in series["columns"]
|
||||||
|
perf = client.post("/performance/assets", json={"symbols": ["S0"], "freq": "weekly"}).json()
|
||||||
|
assert "sharpe" in perf[0]
|
||||||
|
pp = client.post("/performance/portfolio", json={"holdings": {"S0": 10, "S1": 20}}).json()
|
||||||
|
assert "max_drawdown" in pp
|
||||||
|
roll = client.post("/performance/rolling", json={"symbol": "S1", "window": 60}).json()
|
||||||
|
assert "sharpe" in roll["columns"]
|
||||||
|
assert client.post("/risk/assets", json={"symbols": ["NOPE"]}).status_code == 422
|
||||||
|
|
||||||
|
|
||||||
|
def test_models_and_optimize(client):
|
||||||
|
er = client.post("/model/expected-returns", json={"symbols": ["S0", "S1", "S2", "S3"],
|
||||||
|
"views": [{"symbol": "S1", "sentiment": 0.8, "confidence": 0.6}]}).json()
|
||||||
|
assert er["posterior_ann"]["S1"] > er["prior_ann"]["S1"] and len(er["views"]) == 1
|
||||||
|
sc = client.post("/model/scenarios", json={"symbols": ["S0", "S1"], "model": "gbm", "n_scenarios": 500}).json()
|
||||||
|
assert sc["description"]["n_scenarios"] == 512 and sc["description"]["n_requested"] == 500 # Sobol': power of two
|
||||||
|
assert sc["description"]["sampling"] == "sobol" and sc["description"]["moment_matched"] and len(sc["summary"]) == 2
|
||||||
|
assert len(sc["description"]["hash"]) == 16
|
||||||
|
body = {"mode": "new", "cash": 200000, "symbols": ["S0", "S1", "S2", "S3", "S4", "S5"], "cvar_limit": 8000,
|
||||||
|
"n_scenarios": 400, "scenario_model": "gbm", "solver": "highs", "horizon": "week",
|
||||||
|
"importance_sampling": "off"}
|
||||||
|
res = client.post("/optimize", json=body).json()
|
||||||
|
assert res["status"] == "Optimal" and res["solver"] == "highs" and res["reproducible"] is True
|
||||||
|
assert res["cvar"] <= 8000 + 1e-3 and res["cash_end"] >= 0
|
||||||
|
assert all(p["shares"] >= 0 for p in res["positions"])
|
||||||
|
# identical request -> identical scenario hash and identical portfolio
|
||||||
|
again = client.post("/optimize", json=body).json()
|
||||||
|
assert again["scenario_hash"] == res["scenario_hash"] and again["positions"] == res["positions"]
|
||||||
|
# importance sampling and the exact Gaussian mode are accepted and give proposals of the same kind
|
||||||
|
tilted = client.post("/optimize", json={**body, "importance_sampling": "market"}).json()
|
||||||
|
assert tilted["status"] == "Optimal" and tilted["model"]["scenarios"]["weighted"] is True
|
||||||
|
assert tilted["model"]["scenarios"]["importance_sampling"]["direction"] == "market"
|
||||||
|
adaptive = client.post("/optimize", json={**body, "importance_sampling": "adaptive"}).json()
|
||||||
|
assert adaptive["status"] == "Optimal"
|
||||||
|
assert adaptive["solver_info"]["importance_sampling"]["stage1_status"] == "Optimal"
|
||||||
|
# the market tilt does not depend on the risk limit: same scenario key and hash at another limit
|
||||||
|
other = client.post("/optimize", json={**body, "importance_sampling": "market", "cvar_limit": 5000}).json()
|
||||||
|
assert other["scenario_key"]["id"] == tilted["scenario_key"]["id"]
|
||||||
|
assert other["scenario_hash"] == tilted["scenario_hash"] and other["cvar"] <= 5000 + 1e-3
|
||||||
|
exact = client.post("/optimize", json={**body, "scenario_model": "gaussian_exact"}).json()
|
||||||
|
assert exact["status"] == "Optimal" and exact["reproducible"] and exact["scenario_hash"] is None
|
||||||
|
assert exact["solver_info"]["outer_approximation"]["converged"] and exact["cvar"] <= 8000 * (1 + 1e-5)
|
||||||
|
hold = {p["symbol"]: p["shares"] for p in res["positions"]}
|
||||||
|
reb = client.post("/optimize", json={**body, "mode": "rebalance", "holdings": hold, "cash": res["cash_end"],
|
||||||
|
"cvar_limit": 5000}).json()
|
||||||
|
assert reb["status"] == "Optimal" and reb["mode"] == "rebalance"
|
||||||
|
# the starting portfolio is reported on the same scenarios, so before and after are comparable
|
||||||
|
before = reb["before"]
|
||||||
|
assert res["before"] is None # a new portfolio has nothing to compare against
|
||||||
|
assert before["n_positions"] == sum(1 for v in hold.values() if v > 0)
|
||||||
|
assert before["invested"] == pytest.approx(res["invested"], rel=1e-9)
|
||||||
|
assert before["cvar"] > 5000 > reb["cvar"] # the rebalancing is what brings the risk under the new limit
|
||||||
|
assert before["expected_pnl"] > reb["expected_pnl"] # ... at the cost of expected return
|
||||||
|
assert sum(before["sector_exposure"].values()) == pytest.approx(before["invested"] / reb["wealth"], rel=1e-6)
|
||||||
|
|
||||||
|
|
||||||
|
def test_jobs_and_snapshots(client):
|
||||||
|
snaps = client.get("/snapshots").json()
|
||||||
|
assert isinstance(snaps, list)
|
||||||
|
created = client.post("/snapshots", json={"name": "test-snap"})
|
||||||
|
assert created.status_code == 201
|
||||||
|
assert "test-snap" in client.get("/snapshots").json()
|
||||||
|
assert client.get("/jobs").json() == []
|
||||||
|
assert client.get("/jobs/nope").status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_asset_risk_in_value_space(client):
|
||||||
|
body = {"symbols": ["S0", "S1"], "horizon": "month", "confidence": [0.95, 0.99]}
|
||||||
|
log = client.post("/risk/assets", json=body).json()
|
||||||
|
val = client.post("/risk/assets", json={**body, "space": "value"}).json()
|
||||||
|
for a, b in zip(log, val):
|
||||||
|
# a VaR is a quantile: it carries over through 1 - e^(-x) (up to the interpolation between
|
||||||
|
# order statistics, which does not commute with a non-linear map)
|
||||||
|
assert b["var95_hist"] < a["var95_hist"]
|
||||||
|
assert b["var95_hist"] == pytest.approx(1 - np.exp(-a["var95_hist"]), rel=1e-4)
|
||||||
|
assert b["var95_normal_ewm"] == pytest.approx(1 - np.exp(-a["var95_normal_ewm"]), rel=1e-8)
|
||||||
|
# a CVaR is a tail mean: computed in value space it is strictly below the naive conversion
|
||||||
|
for col in ("cvar95_normal_ewm", "cvar99_normal_ewm"):
|
||||||
|
assert b[col] < 1 - np.exp(-a[col])
|
||||||
|
# this fixture drifts upwards, so at a monthly horizon the estimated mean can push the
|
||||||
|
# parametric VaR below zero -- the reason the UI asks for drift="zero"
|
||||||
|
assert a["var95_normal_ewm"] < 0
|
||||||
|
assert b["cvar99_hist"] < 1.0 # a share of the value can never reach 100 %
|
||||||
|
assert b["vol_ewm_ann"] == a["vol_ewm_ann"] # only the loss columns change
|
||||||
|
# the market-risk convention: no drift extrapolated into the horizon
|
||||||
|
zero = client.post("/risk/assets", json={**body, "drift": "zero"}).json()
|
||||||
|
for a, z in zip(log, zero):
|
||||||
|
assert z["var95_normal_ewm"] != a["var95_normal_ewm"]
|
||||||
|
assert z["var95_hist"] == a["var95_hist"] # historical columns keep their window
|
||||||
|
|
||||||
|
|
||||||
|
def test_symbol_table_and_facets(client):
|
||||||
|
t = client.get("/data/symbols", params={"limit": 5}).json()
|
||||||
|
assert t["n_universe"] == 12 and t["n_matched"] == 12 and len(t["symbols"]) == 5
|
||||||
|
caps = [row["market_cap"] for row in t["symbols"]]
|
||||||
|
assert caps == sorted(caps, reverse=True) # largest first
|
||||||
|
assert set(t["facets"]["regions"]) == {"US", "Europe"} and sum(t["facets"]["regions"].values()) == 12
|
||||||
|
row = t["symbols"][0]
|
||||||
|
assert row["price"] > 0 and 0 < row["market_weight"] <= 1 and row["sector"]
|
||||||
|
eu = client.get("/data/symbols", params={"region": "Europe"}).json()
|
||||||
|
assert eu["n_matched"] == 6 and all(r["region"] == "Europe" for r in eu["symbols"])
|
||||||
|
one = client.get("/data/symbols", params={"search": "S1"}).json()
|
||||||
|
assert {r["symbol"] for r in one["symbols"]} >= {"S1", "S10", "S11"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_model_covariance_endpoint(client):
|
||||||
|
body = {"symbols": ["S0", "S1", "S2", "S3"], "freq": "daily", "horizon": "week",
|
||||||
|
"covariance": {"method": "ewm", "lam": 0.99, "shrinkage": "auto"}}
|
||||||
|
c = client.post("/model/covariance", json=body).json()
|
||||||
|
assert c["correlation"]["columns"] == body["symbols"]
|
||||||
|
assert c["correlation"]["data"][0][0] == pytest.approx(1.0)
|
||||||
|
assert 0 <= c["shrinkage_applied"] <= 1 and c["conditioning"]["n_assets"] == 4
|
||||||
|
assert c["conditioning"]["rank_deficient"] is False
|
||||||
|
assert sum(c["eigenvalues"]) == pytest.approx(1.0)
|
||||||
|
# the covariance is scaled to the horizon, the volatility to the year
|
||||||
|
vol_h = c["volatility_horizon"]["S0"]
|
||||||
|
assert c["covariance"]["data"][0][0] == pytest.approx(vol_h ** 2, rel=1e-9)
|
||||||
|
assert c["volatility_ann"]["S0"] == pytest.approx(vol_h * (252 / 5) ** 0.5, rel=1e-9)
|
||||||
|
|
||||||
|
|
||||||
|
def test_scenario_distribution_of_a_portfolio(client):
|
||||||
|
body = {"symbols": ["S0", "S1", "S2"], "model": "gbm", "n_scenarios": 1024, "horizon": "week",
|
||||||
|
"weights": {"S0": 0.5, "S1": 0.3, "S2": 0.2}, "value": 1_000_000, "bins": 30,
|
||||||
|
"focus": ["S0"], "importance_sampling": "off"}
|
||||||
|
d = client.post("/model/scenarios", json=body).json()
|
||||||
|
p = d["portfolio"]
|
||||||
|
assert p["value"] == 1_000_000 and p["invested"] == pytest.approx(1_000_000)
|
||||||
|
assert len(p["histogram"]["mass"]) == 30 and sum(p["histogram"]["mass"]) == pytest.approx(1.0)
|
||||||
|
assert p["risk"]["0.99"]["cvar"] > p["risk"]["0.95"]["cvar"] > p["risk"]["0.95"]["var"]
|
||||||
|
# the quantiles and the VaR are the same statistic seen from two sides (the VaR is the order
|
||||||
|
# statistic the Rockafellar-Uryasev rows use, the quantile interpolates - one rank apart)
|
||||||
|
assert p["quantiles"]["0.05"] == pytest.approx(-p["risk"]["0.95"]["var"], rel=1e-2)
|
||||||
|
assert p["min"] <= p["quantiles"]["0.01"] <= p["quantiles"]["0.5"] <= p["quantiles"]["0.99"] <= p["max"]
|
||||||
|
assert sum(d["asset_histograms"]["S0"]["mass"]) == pytest.approx(1.0)
|
||||||
|
# a tilted set reports the same model quantities, with probability weights
|
||||||
|
tilted = client.post("/model/scenarios", json={**body, "importance_sampling": "market"}).json()
|
||||||
|
assert tilted["description"]["weighted"] is True
|
||||||
|
assert tilted["description"]["effective_sample_size"] < tilted["description"]["n_scenarios"]
|
||||||
|
assert tilted["portfolio"]["risk"]["0.95"]["cvar"] == pytest.approx(p["risk"]["0.95"]["cvar"], rel=0.05)
|
||||||
|
# shares instead of weights
|
||||||
|
shares = client.post("/model/scenarios", json={**body, "weights": {}, "holdings": {"S0": 10, "S1": 5}}).json()
|
||||||
|
assert shares["portfolio"]["invested"] > 0 and len(shares["portfolio"]["exposure"]) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_ui_is_served(client):
|
||||||
|
page = client.get("/ui/")
|
||||||
|
assert page.status_code == 200 and "quantcore" in page.text
|
||||||
|
assert client.get("/ui/js/main.js").status_code == 200
|
||||||
|
assert client.get("/", follow_redirects=False).status_code in (307, 308)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_ui_can_be_switched_off(settings):
|
||||||
|
"""``QUANTCORE_ENABLE_UI=false`` ships the same image with the JSON API and /docs only."""
|
||||||
|
from fastapi import FastAPI
|
||||||
|
|
||||||
|
off = FastAPI()
|
||||||
|
assert app_module.mount_test_bench(off, settings.model_copy(update={"enable_ui": False})) is False
|
||||||
|
with TestClient(off) as c:
|
||||||
|
assert c.get("/ui/").status_code == 404
|
||||||
|
assert c.get("/", follow_redirects=False).status_code == 404
|
||||||
|
|
||||||
|
on = FastAPI()
|
||||||
|
assert app_module.mount_test_bench(on, settings.model_copy(update={"enable_ui": True})) is True
|
||||||
|
with TestClient(on) as c:
|
||||||
|
assert c.get("/ui/").status_code == 200
|
||||||
|
assert c.get("/", follow_redirects=False).headers["location"] == "/ui/"
|
||||||
91
tests/test_black_litterman.py
Normal file
91
tests/test_black_litterman.py
Normal file
@ -0,0 +1,91 @@
|
|||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from quantcore.models.black_litterman import (
|
||||||
|
BlackLittermanModel, View, idzorek_omega, idzorek_omega_numeric, view_from_sentiment,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def model():
|
||||||
|
symbols = ["A", "B", "C", "D"]
|
||||||
|
vols = np.array([0.20, 0.25, 0.15, 0.30]) / np.sqrt(52)
|
||||||
|
corr = np.array([[1, .5, .3, .2], [.5, 1, .4, .1], [.3, .4, 1, .3], [.2, .1, .3, 1]])
|
||||||
|
cov = pd.DataFrame(np.outer(vols, vols) * corr, index=symbols, columns=symbols)
|
||||||
|
w = pd.Series([0.4, 0.3, 0.2, 0.1], index=symbols)
|
||||||
|
return BlackLittermanModel(cov, w, market_return=0.07 / 52, risk_free=0.01 / 52, tau=0.05, periods_per_year=52)
|
||||||
|
|
||||||
|
|
||||||
|
def test_equilibrium_is_consistent(model):
|
||||||
|
# the market portfolio earns exactly the market excess return under the implied returns
|
||||||
|
assert model.w @ model.pi == pytest.approx(model.market_return - model.risk_free)
|
||||||
|
assert model.delta > 0
|
||||||
|
res = model.posterior([])
|
||||||
|
pd.testing.assert_series_equal(res.posterior, res.prior)
|
||||||
|
# market weights are recovered as the unconstrained MV weights of the prior (tau-adjusted cov ignored here)
|
||||||
|
w_mv = np.linalg.solve(model.delta * model.sigma, model.pi)
|
||||||
|
np.testing.assert_allclose(w_mv, model.w)
|
||||||
|
|
||||||
|
|
||||||
|
def test_idzorek_closed_form_equals_numeric(model):
|
||||||
|
p = np.array([0.0, 1.0, 0.0, 0.0])
|
||||||
|
q = (0.12 - 0.01) / 52
|
||||||
|
for c in (0.1, 0.35, 0.5, 0.8, 0.99):
|
||||||
|
omega_cf = idzorek_omega(p, model.tau * model.sigma, c)
|
||||||
|
omega_num = idzorek_omega_numeric(p, q, model.pi, model.sigma, model.tau, model.delta, c)
|
||||||
|
assert omega_cf == pytest.approx(omega_num, rel=1e-4)
|
||||||
|
|
||||||
|
|
||||||
|
def test_confidence_moves_weights_by_fraction(model):
|
||||||
|
view = View({"B": 1.0}, q=0.15, confidence=0.4)
|
||||||
|
res = model.posterior([view])
|
||||||
|
w_post = np.linalg.solve(model.delta * model.sigma, res.posterior.values)
|
||||||
|
full = model.posterior([View({"B": 1.0}, q=0.15, confidence=1.0)])
|
||||||
|
w_full = np.linalg.solve(model.delta * model.sigma, full.posterior.values)
|
||||||
|
tilt_partial = w_post - model.w
|
||||||
|
tilt_full = w_full - model.w
|
||||||
|
np.testing.assert_allclose(tilt_partial, 0.4 * tilt_full, atol=1e-10)
|
||||||
|
|
||||||
|
|
||||||
|
def test_full_confidence_reproduces_view(model):
|
||||||
|
res = model.posterior([View({"B": 1.0}, q=0.15, confidence=1.0)])
|
||||||
|
assert res.posterior_ann["B"] == pytest.approx(0.15, abs=1e-9)
|
||||||
|
|
||||||
|
|
||||||
|
def test_positive_view_raises_asset_and_relative_view(model):
|
||||||
|
res = model.posterior([View({"A": 1.0}, q=0.20, confidence=0.6)])
|
||||||
|
assert res.posterior_ann["A"] > res.prior_ann["A"]
|
||||||
|
rel = model.posterior([View({"C": 1.0, "D": -1.0}, q=0.05, confidence=0.5, label="C beats D")])
|
||||||
|
assert (rel.posterior_ann["C"] - rel.posterior_ann["D"]) > (rel.prior_ann["C"] - rel.prior_ann["D"])
|
||||||
|
assert rel.views[0]["label"] == "C beats D"
|
||||||
|
assert rel.posterior_cov.shape == (4, 4)
|
||||||
|
|
||||||
|
|
||||||
|
def test_view_from_sentiment(model):
|
||||||
|
v = view_from_sentiment(model, "C", sentiment=1.0, confidence=0.7, strength=0.25)
|
||||||
|
assert v.q == pytest.approx(model.prior_ann["C"] + 0.25 * 0.15, rel=1e-6)
|
||||||
|
neg = view_from_sentiment(model, "C", sentiment=-1.0, confidence=0.7)
|
||||||
|
assert neg.q < model.prior_ann["C"]
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
View({"A": 1.0}, q=0.1, confidence=0.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_documentation_example_numbers(model):
|
||||||
|
"""docs/measurement_and_models.tex, Example 7.2 (the fixture is that four-asset model)."""
|
||||||
|
assert model.market_var * 52 == pytest.approx(0.02502, abs=1e-5)
|
||||||
|
assert model.delta == pytest.approx(2.3986, abs=1e-4)
|
||||||
|
np.testing.assert_allclose(model.prior_ann.values, [0.0736, 0.0880, 0.0435, 0.0550], atol=5e-5)
|
||||||
|
assert model.w @ model.prior_ann.values == pytest.approx(0.07, abs=1e-9)
|
||||||
|
res = model.posterior([View({"B": 1.0}, q=0.15, confidence=0.4)])
|
||||||
|
p = np.array([0.0, 1.0, 0.0, 0.0])
|
||||||
|
base = p @ (model.tau * model.sigma) @ p
|
||||||
|
assert base == pytest.approx(6.0096e-5, rel=1e-4)
|
||||||
|
assert res.omega[0] == pytest.approx(1.5 * base, rel=1e-12)
|
||||||
|
assert res.omega[0] == pytest.approx(9.0144e-5, rel=1e-4)
|
||||||
|
np.testing.assert_allclose(res.posterior_ann.values, [0.0835, 0.1128, 0.0494, 0.0580], atol=5e-5)
|
||||||
|
w_post = np.linalg.solve(model.delta * model.sigma, res.posterior.values)
|
||||||
|
full = model.posterior([View({"B": 1.0}, q=0.15, confidence=1.0)])
|
||||||
|
w_100 = np.linalg.solve(model.delta * model.sigma, full.posterior.values)
|
||||||
|
assert w_100[1] == pytest.approx(0.7139, abs=5e-5) and w_post[1] == pytest.approx(0.4656, abs=5e-5)
|
||||||
|
np.testing.assert_allclose(w_post[[0, 2, 3]], model.w[[0, 2, 3]], atol=1e-10)
|
||||||
132
tests/test_cleaning.py
Normal file
132
tests/test_cleaning.py
Normal file
@ -0,0 +1,132 @@
|
|||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from quantcore.data.cleaning import (
|
||||||
|
CleaningReport,
|
||||||
|
clean_fx,
|
||||||
|
clean_prices,
|
||||||
|
find_spike_blocks,
|
||||||
|
fx_scale_exponent,
|
||||||
|
longest_missing_run,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_spike_blocks_and_genuine_moves():
|
||||||
|
idx = pd.bdate_range("2024-01-01", periods=40)
|
||||||
|
p = pd.Series(100.0, index=idx)
|
||||||
|
p.iloc[5] = 200.0 # single wrong tick (2x, reverses next day)
|
||||||
|
p.iloc[10:12] = 210.0 # two-day block
|
||||||
|
p.iloc[20:] = 40.0 # genuine crash that stays
|
||||||
|
p.iloc[30:33] = 45.0 # ordinary small moves
|
||||||
|
assert find_spike_blocks(p) == [5, 10, 11]
|
||||||
|
|
||||||
|
|
||||||
|
def test_clean_prices_report_and_gaps():
|
||||||
|
idx = pd.bdate_range("2024-01-01", periods=40)
|
||||||
|
a = pd.Series(np.linspace(50, 60, 40), index=idx)
|
||||||
|
b = a.copy()
|
||||||
|
b.iloc[3:8] = -1.0 # broken adjustment stretch
|
||||||
|
b.iloc[25] = 300.0 # spike
|
||||||
|
from quantcore.data.cleaning import CleaningReport
|
||||||
|
|
||||||
|
rep = CleaningReport()
|
||||||
|
out = clean_prices(pd.DataFrame({"A": a, "B": b}), report=rep)
|
||||||
|
assert rep.n_nonpositive == 5 and rep.symbols_nonpositive == {"B": 5}
|
||||||
|
assert rep.n_spikes == 1 and rep.symbols_spikes == {"B": 1}
|
||||||
|
assert out["A"].notna().all()
|
||||||
|
assert out["B"].isna().sum() == 6
|
||||||
|
runs = longest_missing_run(out)
|
||||||
|
assert runs["A"] == 0 and runs["B"] == 5
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# FX quoting-unit changes (the NOK/CHF artefact: 96 observations returned 100x too small)
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
def _fx_series(n=500, seed=0):
|
||||||
|
rng = np.random.default_rng(seed)
|
||||||
|
idx = pd.bdate_range("2016-01-01", periods=n)
|
||||||
|
return pd.Series(0.12 * np.exp(np.cumsum(rng.standard_normal(n) * 0.004)), index=idx)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_quoting_unit_change_is_undone_wherever_it_sits():
|
||||||
|
"""A stretch quoted 100x too small is repaired, whatever part of the series it covers."""
|
||||||
|
good = _fx_series()
|
||||||
|
for lo, hi in ((70, 140), (0, 60), (440, 500)): # middle, start, end
|
||||||
|
broken = good.copy()
|
||||||
|
broken.iloc[lo:hi] /= 100.0
|
||||||
|
fixed = clean_fx(broken.to_frame("NOK"))["NOK"]
|
||||||
|
assert np.allclose(fixed.values, good.values), (lo, hi)
|
||||||
|
# the artefact is what made the volatility absurd; the repair takes it back
|
||||||
|
vol = lambda x: float(np.log(x).diff().std() * np.sqrt(252))
|
||||||
|
assert vol(broken) > 2.0 and vol(fixed) == pytest.approx(vol(good), rel=1e-9)
|
||||||
|
|
||||||
|
|
||||||
|
def test_several_unit_changes_and_other_powers_of_ten():
|
||||||
|
good = _fx_series()
|
||||||
|
broken = good.copy()
|
||||||
|
broken.iloc[30:60] /= 100.0
|
||||||
|
broken.iloc[200:210] *= 10.0
|
||||||
|
broken.iloc[300:301] /= 1000.0
|
||||||
|
fixed = clean_fx(broken.to_frame("NOK"))["NOK"]
|
||||||
|
assert np.allclose(fixed.values, good.values)
|
||||||
|
assert set(fx_scale_exponent(broken).unique()) == {0, -2, 1, -3}
|
||||||
|
|
||||||
|
|
||||||
|
def test_genuine_moves_are_never_rescaled():
|
||||||
|
"""A devaluation is a price move, not a unit change: nothing short of a factor of ten is touched.
|
||||||
|
|
||||||
|
The 6.1x case is real: ABVX.PA gained 510 % on its trial results (2025-07-23). A loose tolerance
|
||||||
|
reads that as a decimal shift and rewrites the year of history that follows it.
|
||||||
|
"""
|
||||||
|
good = _fx_series()
|
||||||
|
for factor in (0.75, 1.4, 0.5, 2.0, 3.0, 6.101, 6.421, 1 / 6.101, 8.0):
|
||||||
|
moved = good.copy()
|
||||||
|
moved.iloc[250:] *= factor
|
||||||
|
fixed = clean_fx(moved.to_frame("TRY"))["TRY"]
|
||||||
|
assert np.allclose(fixed.values, moved.values), factor
|
||||||
|
|
||||||
|
|
||||||
|
def test_clean_fx_is_idempotent_and_reports_per_currency():
|
||||||
|
good = _fx_series()
|
||||||
|
frame = pd.DataFrame({"CHF": 1.0, "EUR": good * 8, "NOK": good})
|
||||||
|
frame.iloc[100:180, frame.columns.get_loc("NOK")] /= 100.0
|
||||||
|
report = CleaningReport()
|
||||||
|
fixed = clean_fx(frame, report=report)
|
||||||
|
assert report.n_fx_rescaled == 80 and report.fx_rescaled == {"NOK": 80}
|
||||||
|
assert np.allclose(fixed["NOK"].values, good.values)
|
||||||
|
assert fixed["EUR"].equals(frame["EUR"]) and fixed["CHF"].equals(frame["CHF"])
|
||||||
|
again = CleaningReport()
|
||||||
|
assert clean_fx(fixed, report=again).equals(fixed) and again.n_fx_rescaled == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_short_series_and_missing_values_are_handled():
|
||||||
|
idx = pd.bdate_range("2024-01-01", periods=6)
|
||||||
|
assert fx_scale_exponent(pd.Series([0.1], index=idx[:1])).empty is False
|
||||||
|
gappy = pd.Series([0.1, np.nan, 0.001, 0.001, np.nan, 0.1], index=idx)
|
||||||
|
fixed = clean_fx(gappy.to_frame("NOK"))["NOK"]
|
||||||
|
assert fixed.dropna().round(6).tolist() == [0.1, 0.1, 0.1, 0.1]
|
||||||
|
assert fixed.isna().tolist() == [False, True, False, False, True, False]
|
||||||
|
|
||||||
|
|
||||||
|
def test_fx_spikes_are_removed_but_a_real_regime_break_is_kept():
|
||||||
|
"""A reversing tick is a quote error; 2015-01-16 (the SNB floor) is not, and must survive."""
|
||||||
|
good = _fx_series(n=400)
|
||||||
|
broken = good.copy()
|
||||||
|
broken.iloc[120] *= 1.09 # one-day spike, reverses next day
|
||||||
|
broken.iloc[250:] *= 0.81 # a permanent 19 % repricing: never reversed
|
||||||
|
fixed = clean_fx(broken.to_frame("EUR"))["EUR"]
|
||||||
|
assert np.isnan(fixed.iloc[120]) # the tick is gone
|
||||||
|
assert fixed.iloc[249] == broken.iloc[249] and fixed.iloc[250] == broken.iloc[250]
|
||||||
|
assert fixed.drop(fixed.index[120]).equals(broken.drop(broken.index[120]))
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_unit_repair_runs_before_the_spike_pass():
|
||||||
|
"""A 100x stretch must be rescaled, not mistaken for two enormous spikes at its edges."""
|
||||||
|
good = _fx_series(n=300)
|
||||||
|
broken = good.copy()
|
||||||
|
broken.iloc[100:160] /= 100.0
|
||||||
|
report = CleaningReport()
|
||||||
|
fixed = clean_fx(broken.to_frame("NOK"), report=report)["NOK"]
|
||||||
|
assert report.n_fx_rescaled == 60 and report.n_fx_spikes == 0
|
||||||
|
assert np.allclose(fixed.values, good.values)
|
||||||
222
tests/test_client.py
Normal file
222
tests/test_client.py
Normal file
@ -0,0 +1,222 @@
|
|||||||
|
"""The HTTP client and the ``quantcore-api`` CLI, against the app running in this process.
|
||||||
|
|
||||||
|
``TestClient`` satisfies the transport contract of :class:`QuantcoreClient` (``request(method, url,
|
||||||
|
params=, json=, timeout=)``), so the same code paths that talk to the Spark are exercised here
|
||||||
|
without a server: the URLs, the query parameters, the request bodies and the error handling.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
import quantcore.api.app as app_module
|
||||||
|
from quantcore import apicli
|
||||||
|
from quantcore.client import QuantcoreClient, QuantcoreError
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def api(settings, store):
|
||||||
|
app_module.state = app_module.State(settings)
|
||||||
|
with TestClient(app_module.app) as transport:
|
||||||
|
yield QuantcoreClient("http://testserver", session=transport)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def cli(api, monkeypatch):
|
||||||
|
"""Run the CLI against the in-process app: every command builds its client through ``_client``."""
|
||||||
|
monkeypatch.setattr(apicli, "_client", lambda args: api)
|
||||||
|
return apicli.main
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# client
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
def test_the_client_covers_the_api(api):
|
||||||
|
health = api.health()
|
||||||
|
assert health["status"] == "ok" and health["store"]["prices"] is True
|
||||||
|
|
||||||
|
assert len(api.universe(region="Europe")) == 6
|
||||||
|
assert api.symbol("S0")["symbol"] == "S0"
|
||||||
|
table = api.symbols_table(limit=5)
|
||||||
|
assert len(table["symbols"]) == 5 and "sectors" in table["facets"]
|
||||||
|
assert table["n_universe"] >= table["n_matched"] >= 5
|
||||||
|
|
||||||
|
px = api.prices(["S0", "S1"], freq="weekly")
|
||||||
|
assert px["columns"] == ["S0", "S1"] and px["freq"] == "weekly"
|
||||||
|
assert api.returns(["S0"])["kind"] == "log_returns"
|
||||||
|
assert api.market_index(freq="weekly")["n"] > 100
|
||||||
|
assert sum(api.market_weights()["weights"].values()) == pytest.approx(1.0)
|
||||||
|
|
||||||
|
rows = api.asset_risk(["S0", "S1"], horizon="week")
|
||||||
|
assert len(rows) == 2 and rows[0]["cvar95_hist"] > rows[0]["var95_hist"]
|
||||||
|
assert api.risk_series("S0", horizon="week")["index"]
|
||||||
|
assert api.covariance(["S0", "S1"])["columns"] == ["S0", "S1"]
|
||||||
|
assert api.correlation(["S0", "S1"])["data"][0][0] == pytest.approx(1.0)
|
||||||
|
assert api.portfolio_risk({"S0": 100, "S1": 50}, cash=1000.0, horizon="week")["value"] > 0
|
||||||
|
|
||||||
|
assert len(api.asset_performance(["S0", "S1"])) == 2
|
||||||
|
assert api.rolling_performance("S0", window=252)["index"]
|
||||||
|
assert api.portfolio_performance({"S0": 100})
|
||||||
|
|
||||||
|
assert api.expected_returns(max_assets=6)["market_return_ann"] != 0
|
||||||
|
assert api.model_covariance(["S0", "S1"])["conditioning"]["n_assets"] == 2
|
||||||
|
scen = api.scenarios(["S0", "S1"], model="gbm", n_scenarios=1024, holdings={"S0": 10})
|
||||||
|
assert scen["description"]["n_scenarios"] == 1024 and scen["portfolio"]["invested"] > 0
|
||||||
|
|
||||||
|
# order-independent: another module may have created a snapshot on the session-scoped store
|
||||||
|
assert isinstance(api.jobs(), list) and isinstance(api.snapshots(), list)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_client_optimises(api):
|
||||||
|
res = api.optimize(cash=1_000_000, max_assets=6, horizon="week", scenario_model="gaussian_exact",
|
||||||
|
solver="highs", time_limit=120)
|
||||||
|
assert res["status"] in ("Optimal", "FeasibleFound")
|
||||||
|
assert res["solver"] == "highs" and res["positions"]
|
||||||
|
# a rebalancing run carries the starting point, so before/after can be compared
|
||||||
|
holdings = {p["symbol"]: p["shares"] for p in res["positions"][:2]}
|
||||||
|
reb = api.optimize(mode="rebalance", holdings=holdings, cash=10_000, max_assets=6, horizon="week",
|
||||||
|
scenario_model="gaussian_exact", solver="highs", time_limit=120)
|
||||||
|
assert reb["before"] is not None and reb["before"]["invested"] > 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_none_arguments_are_left_to_the_api(api):
|
||||||
|
"""Optional arguments that are ``None`` must not be sent, or the API would see "None"."""
|
||||||
|
sent: dict = {}
|
||||||
|
original = api.session.request
|
||||||
|
|
||||||
|
def spy(method, url, **kw):
|
||||||
|
sent.update({"params": kw.get("params"), "json": kw.get("json")})
|
||||||
|
return original(method, url, **kw)
|
||||||
|
|
||||||
|
api.session.request = spy
|
||||||
|
try:
|
||||||
|
api.prices(["S0"], start=None, end=None)
|
||||||
|
assert "start" not in (sent["params"] or {}) and "end" not in (sent["params"] or {})
|
||||||
|
api.expected_returns(symbols=None, max_assets=6)
|
||||||
|
assert "symbols" not in (sent["json"] or {})
|
||||||
|
finally:
|
||||||
|
api.session.request = original
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_client_reports_api_errors(api):
|
||||||
|
with pytest.raises(QuantcoreError) as exc:
|
||||||
|
api.symbol("NOPE")
|
||||||
|
assert exc.value.status_code == 404 and not exc.value.unreachable
|
||||||
|
|
||||||
|
with pytest.raises(QuantcoreError) as exc: # 422: the field name must reach the caller
|
||||||
|
api.asset_risk(["S0"], window=1)
|
||||||
|
assert exc.value.status_code == 422 and "window" in str(exc.value)
|
||||||
|
assert "greater than or equal to 10" in str(exc.value)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_client_reports_an_unreachable_server():
|
||||||
|
with QuantcoreClient("http://127.0.0.1:9", timeout=2.0) as qc, pytest.raises(QuantcoreError) as exc:
|
||||||
|
qc.health()
|
||||||
|
assert exc.value.unreachable and "127.0.0.1:9" in str(exc.value)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_client_retries_dropped_gets_but_never_posts():
|
||||||
|
"""A poll on a kept-alive connection the server just closed must not abort `update --wait`;
|
||||||
|
a POST (optimisation, job start) must never be sent twice."""
|
||||||
|
with QuantcoreClient("http://127.0.0.1:9") as qc:
|
||||||
|
retry = qc.session.get_adapter("http://127.0.0.1:9/health").max_retries
|
||||||
|
assert retry.read >= 1 and "GET" in retry.allowed_methods and "POST" not in retry.allowed_methods
|
||||||
|
|
||||||
|
|
||||||
|
def test_wait_for_job_polls_until_the_job_ends():
|
||||||
|
states = iter(["queued", "running", "running", "done"])
|
||||||
|
|
||||||
|
class Stub:
|
||||||
|
def request(self, method, url, **kw):
|
||||||
|
class Res:
|
||||||
|
status_code = 200
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def json():
|
||||||
|
return {"job_id": "abc", "status": next(states)}
|
||||||
|
return Res()
|
||||||
|
|
||||||
|
qc = QuantcoreClient("http://stub", session=Stub())
|
||||||
|
seen: list[str] = []
|
||||||
|
record = qc.wait_for_job("abc", poll=0.0, on_poll=lambda r: seen.append(r["status"]))
|
||||||
|
assert record["status"] == "done" and seen == ["queued", "running", "running"]
|
||||||
|
|
||||||
|
with pytest.raises(QuantcoreError): # a job that never finishes hits the deadline
|
||||||
|
QuantcoreClient("http://stub", session=type("S", (), {
|
||||||
|
"request": lambda self, m, u, **k: type("R", (), {
|
||||||
|
"status_code": 200, "json": staticmethod(lambda: {"status": "running"})})()})()
|
||||||
|
).wait_for_job("abc", poll=0.0, deadline=-1.0)
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
# CLI
|
||||||
|
# ------------------------------------------------------------------------------------------------
|
||||||
|
def test_every_documented_command_parses():
|
||||||
|
parser = apicli.build_parser()
|
||||||
|
for argv in (
|
||||||
|
["health"], ["endpoints"], ["universe", "--region", "Europe", "--sector", "Health Care"],
|
||||||
|
["symbol", "S0"], ["prices", "S0", "S1", "--freq", "weekly", "--tail", "5"],
|
||||||
|
["returns", "S0", "--currency", "local"], ["market", "weights", "--max-assets", "10"],
|
||||||
|
["risk", "S0", "S1", "--horizon", "week", "--drift", "zero", "--space", "value"],
|
||||||
|
["risk-portfolio", "--holdings", '{"S0": 10}'], ["correlation", "S0", "S1"],
|
||||||
|
["covariance", "S0", "S1", "--method", "rolling"], ["performance", "S0"],
|
||||||
|
["expected-returns", "--max-assets", "50", "--view", "S0:0.6:0.4"],
|
||||||
|
["model-covariance", "S0", "S1"], ["scenarios", "S0", "--model", "gbm", "--n-scenarios", "1024"],
|
||||||
|
["optimize", "--cash", "1000000", "--cvar-limit", "40000", "--risk-drift", "zero"],
|
||||||
|
["optimize", "--request", "r.json", "--json"], ["jobs", "--job-id", "abc"],
|
||||||
|
["setup", "--offline", "--wait"], ["update", "--full", "--wait"],
|
||||||
|
["snapshots", "--create", "--name", "demo"], ["selftest", "--full", "--write"],
|
||||||
|
):
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
assert callable(args.func), argv
|
||||||
|
|
||||||
|
|
||||||
|
def test_sentiment_views_are_parsed():
|
||||||
|
assert apicli._views(["S0:0.6"]) == [{"symbol": "S0", "sentiment": 0.6}]
|
||||||
|
assert apicli._views(["S0:0.6:0.4:0.3"]) == [
|
||||||
|
{"symbol": "S0", "sentiment": 0.6, "confidence": 0.4, "strength": 0.3}]
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
apicli._views(["S0"])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("argv", [
|
||||||
|
["health"], ["endpoints"], ["universe", "--region", "Europe"], ["symbol", "S0"],
|
||||||
|
["prices", "S0", "S1", "--tail", "3"], ["returns", "S0", "--tail", "3"], ["market", "index"],
|
||||||
|
["market", "weights"], ["risk", "S0", "S1", "--drift", "zero"],
|
||||||
|
["risk-portfolio", "--holdings", '{"S0": 10}'], ["correlation", "S0", "S1"],
|
||||||
|
["covariance", "S0", "S1"], ["performance", "S0"], ["expected-returns", "--max-assets", "6"],
|
||||||
|
["model-covariance", "S0", "S1"], ["scenarios", "S0", "S1", "--n-scenarios", "1024"],
|
||||||
|
["jobs"], ["snapshots"],
|
||||||
|
])
|
||||||
|
def test_the_cli_runs_every_read_only_command(cli, capsys, argv):
|
||||||
|
assert cli(argv) == 0
|
||||||
|
assert capsys.readouterr().out.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_cli_optimises_and_prints_a_summary(cli, capsys):
|
||||||
|
assert cli(["optimize", "--max-assets", "6", "--scenario-model", "gaussian_exact",
|
||||||
|
"--solver", "highs", "--time-limit", "120"]) == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "status Optimal" in out and "E[P&L]" in out and "CVaR" in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_selftest_passes_against_a_healthy_service(cli, capsys):
|
||||||
|
"""The selftest is the deployment check: every endpoint, one table, exit 0 when all answered."""
|
||||||
|
assert cli(["selftest", "--n-symbols", "2"]) == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "FAIL" not in out
|
||||||
|
assert "0 failed" in out and "3 skipped" in out # the writing endpoints stay untouched
|
||||||
|
for endpoint in ("GET /health", "POST /risk/portfolio", "POST /optimize", "GET /snapshots"):
|
||||||
|
assert endpoint in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_selftest_fails_loudly_when_the_service_is_down(capsys, monkeypatch):
|
||||||
|
monkeypatch.setattr(apicli, "_client", lambda args: QuantcoreClient("http://127.0.0.1:9", timeout=1.0))
|
||||||
|
assert apicli.main(["selftest"]) == 1
|
||||||
|
assert "not usable" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_cli_exits_non_zero_when_the_api_is_unreachable(capsys):
|
||||||
|
assert apicli.main(["--url", "http://127.0.0.1:9", "--timeout", "1", "health"]) == 1
|
||||||
|
assert "is the service running?" in capsys.readouterr().err
|
||||||
81
tests/test_data.py
Normal file
81
tests/test_data.py
Normal file
@ -0,0 +1,81 @@
|
|||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from quantcore.data.download_service import splice_adjusted
|
||||||
|
from quantcore.data.fx import FXConverter
|
||||||
|
from quantcore.data.yahoo import apply_price_scale, normalise_info
|
||||||
|
from quantcore.universe.sources import parse_ishares_holdings
|
||||||
|
from quantcore.universe.symbols import normalise_share_class_name, to_yahoo_symbol
|
||||||
|
|
||||||
|
|
||||||
|
def test_fx_conversion_and_log_return_identity():
|
||||||
|
idx = pd.bdate_range("2024-01-01", periods=5)
|
||||||
|
fx = pd.DataFrame({"USD": [0.85, 0.86, 0.84, 0.85, 0.87], "CHF": 1.0}, index=idx)
|
||||||
|
conv = FXConverter(fx, base="CHF")
|
||||||
|
px = pd.DataFrame({"A": [10.0, 11.0, 10.5, 10.8, 11.2], "B": [100.0, 101.0, 99.0, 98.0, 100.0]}, index=idx)
|
||||||
|
cur = pd.Series({"A": "USD", "B": "CHF"})
|
||||||
|
chf = conv.convert_prices(px, cur)
|
||||||
|
np.testing.assert_allclose(chf["A"], px["A"] * fx["USD"])
|
||||||
|
np.testing.assert_allclose(chf["B"], px["B"])
|
||||||
|
# CHF log-return = local log-return + FX log-return
|
||||||
|
r_chf = np.log(chf).diff().iloc[1:]
|
||||||
|
r_loc = np.log(px).diff().iloc[1:]
|
||||||
|
r_fx = conv.fx_log_returns(cur, idx).iloc[1:]
|
||||||
|
np.testing.assert_allclose(r_chf.values, (r_loc + r_fx).values)
|
||||||
|
with pytest.raises(KeyError):
|
||||||
|
conv.convert_prices(px, pd.Series({"A": "JPY", "B": "CHF"}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_gbp_pence_normalisation():
|
||||||
|
info = normalise_info("HSBA.L", {"currency": "GBp", "marketCap": 1e11, "regularMarketPrice": 1533.8,
|
||||||
|
"firstTradeDateMilliseconds": 584089200000, "quoteType": "EQUITY"})
|
||||||
|
assert info["currency"] == "GBP" and info["price_scale"] == 100.0
|
||||||
|
assert info["last_price"] == pytest.approx(15.338)
|
||||||
|
assert info["first_trade_date"] == "1988-07-05"
|
||||||
|
px = pd.DataFrame({"HSBA.L": [1500.0, 1600.0], "NESN.SW": [80.0, 81.0]})
|
||||||
|
scaled = apply_price_scale(px, pd.Series({"HSBA.L": 100.0}))
|
||||||
|
assert scaled["HSBA.L"].iloc[0] == 15.0 and scaled["NESN.SW"].iloc[1] == 81.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_splice_rescales_history_on_dividend():
|
||||||
|
idx = pd.bdate_range("2024-01-01", periods=10)
|
||||||
|
stored = pd.DataFrame({"A": np.linspace(100, 109, 10), "B": np.linspace(50, 59, 10)}, index=idx)
|
||||||
|
fresh_idx = pd.bdate_range("2024-01-08", periods=7) # overlaps the last 5 stored days, adds 2 new
|
||||||
|
fresh = pd.DataFrame({"A": np.concatenate([stored["A"].iloc[5:].values * 0.98, [110 * 0.98, 111 * 0.98]]),
|
||||||
|
"B": np.concatenate([stored["B"].iloc[5:].values, [60.0, 61.0]])}, index=fresh_idx)
|
||||||
|
out, rescaled = splice_adjusted(stored, fresh)
|
||||||
|
assert rescaled == ["A"]
|
||||||
|
assert len(out) == 12
|
||||||
|
np.testing.assert_allclose(out["A"].iloc[:5], stored["A"].iloc[:5] * 0.98) # history rescaled
|
||||||
|
np.testing.assert_allclose(out["A"].iloc[5:], fresh["A"]) # fresh window authoritative
|
||||||
|
np.testing.assert_allclose(out["B"], np.concatenate([stored["B"].values, [60.0, 61.0]]))
|
||||||
|
|
||||||
|
|
||||||
|
def test_symbol_mapping_rules():
|
||||||
|
assert to_yahoo_symbol("NOVO B", "Omx Nordic Exchange Copenhagen A/S") == "NOVO-B.CO"
|
||||||
|
assert to_yahoo_symbol("BT.A", "London Stock Exchange") == "BT-A.L"
|
||||||
|
assert to_yahoo_symbol("RR.", "London Stock Exchange") == "RR.L"
|
||||||
|
assert to_yahoo_symbol("NESN", "SIX Swiss Exchange") == "NESN.SW"
|
||||||
|
assert to_yahoo_symbol("BRK.B", country="United States") == "BRK-B"
|
||||||
|
assert to_yahoo_symbol("NDA FI", "Nasdaq Omx Helsinki Ltd.") == "NDA-FI.HE"
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
to_yahoo_symbol("X", "Mars Exchange")
|
||||||
|
assert normalise_share_class_name("Alphabet Inc. Class A") == normalise_share_class_name("Alphabet Inc. Class C")
|
||||||
|
assert normalise_share_class_name("Investor AB ser. B") == normalise_share_class_name("Investor Class A")
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_ishares_holdings():
|
||||||
|
text = ('Fondsposition per,"18.Sept.2026"\n \nEmittententicker,Name,Sektor,Anlageklasse,Marktwert,'
|
||||||
|
'Gewichtung (%),Nominalwert,Nominale,Kurs,Standort,Börse,Marktwährung\n'
|
||||||
|
'"ASML","ASML HOLDING","IT","Aktien","438.829.188,00","4,23","438.829.188,00","303.478,00","1.446,00",'
|
||||||
|
'"Niederlande","Euronext Amsterdam","EUR"\n'
|
||||||
|
'"NOVO B","NOVO NORDISK CLASS B","Gesundheitsversorgung","Aktien","1,00","1,00","1,00","1,00","1,00",'
|
||||||
|
'"Dänemark","Omx Nordic Exchange Copenhagen A/S","DKK"\n'
|
||||||
|
'"XEUR","EUR CASH","Barmittel und/oder Derivate","Geldmarkt","1,00","0,10","1,00","1,00","1,00",'
|
||||||
|
'"Deutschland","-","EUR"\n')
|
||||||
|
df = parse_ishares_holdings(text)
|
||||||
|
assert list(df["symbol"]) == ["ASML.AS", "NOVO-B.CO"]
|
||||||
|
assert df.iloc[0]["gics_sector"] == "Information Technology" and df.iloc[0]["country"] == "Netherlands"
|
||||||
|
assert df.iloc[0]["index_weight"] == pytest.approx(4.23)
|
||||||
|
assert df.iloc[0]["source_date"] == "2026-09-18"
|
||||||
54
tests/test_estimators.py
Normal file
54
tests/test_estimators.py
Normal file
@ -0,0 +1,54 @@
|
|||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from quantcore.measurement import estimators as est
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def r():
|
||||||
|
rng = np.random.default_rng(3)
|
||||||
|
x = rng.standard_normal((600, 4)) @ np.array([[1, .5, .2, 0], [0, 1, .3, 0], [0, 0, 1, .4], [0, 0, 0, 1]]) * 0.01
|
||||||
|
return pd.DataFrame(x, index=pd.bdate_range("2020-01-01", periods=600), columns=list("ABCD"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_rolling_cov_matches_numpy(r):
|
||||||
|
cov = est.rolling_cov(r, 100)
|
||||||
|
np.testing.assert_allclose(cov.values, np.cov(r.values[-100:], rowvar=False, ddof=1))
|
||||||
|
corr = est.rolling_corr(r, 100)
|
||||||
|
np.testing.assert_allclose(np.diag(corr.values), 1.0)
|
||||||
|
np.testing.assert_allclose(corr.values, np.corrcoef(r.values[-100:], rowvar=False), atol=1e-12)
|
||||||
|
|
||||||
|
|
||||||
|
def test_ewm_cov_diagonal_matches_pandas(r):
|
||||||
|
lam = 0.94
|
||||||
|
cov = est.ewm_cov(r, lam, max_obs=len(r))
|
||||||
|
pd_var = r.ewm(alpha=1 - lam, adjust=True).var(bias=True).iloc[-1]
|
||||||
|
np.testing.assert_allclose(np.diag(cov.values), pd_var.values, rtol=1e-8)
|
||||||
|
assert est.ewm_weights(10, lam).sum() == pytest.approx(1.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_ewm_sequence_recursion_converges_to_direct(r):
|
||||||
|
lam = 0.97
|
||||||
|
last = None
|
||||||
|
for date, cov_prev, _ in est.covariance_sequence(r, "ewm", lam=lam, warmup=100):
|
||||||
|
last = (date, cov_prev)
|
||||||
|
# Sigma_{T-1} from the recursion vs. the direct (normalised-weight) estimate at T-1
|
||||||
|
direct = est.ewm_cov(r.iloc[:-1], lam, max_obs=len(r))
|
||||||
|
assert last[0] == r.index[-1]
|
||||||
|
np.testing.assert_allclose(last[1], direct.values, rtol=0.05, atol=1e-7)
|
||||||
|
|
||||||
|
|
||||||
|
def test_rolling_sequence_matches_cov(r):
|
||||||
|
seq = list(est.covariance_sequence(r, "rolling", window=50))
|
||||||
|
date, cov_prev, ret = seq[-1]
|
||||||
|
assert date == r.index[-1]
|
||||||
|
np.testing.assert_allclose(cov_prev, np.cov(r.values[-51:-1], rowvar=False, ddof=1))
|
||||||
|
np.testing.assert_allclose(ret, r.values[-1])
|
||||||
|
|
||||||
|
|
||||||
|
def test_shrinkage_keeps_variances(r):
|
||||||
|
cov = est.rolling_cov(r, 200)
|
||||||
|
sh = est.shrink(cov, 0.3)
|
||||||
|
np.testing.assert_allclose(np.diag(sh.values), np.diag(cov.values))
|
||||||
|
assert abs(sh.values[0, 1]) == pytest.approx(0.7 * abs(cov.values[0, 1]))
|
||||||
166
tests/test_problem.py
Normal file
166
tests/test_problem.py
Normal file
@ -0,0 +1,166 @@
|
|||||||
|
import itertools
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from quantcore.config import Settings
|
||||||
|
from quantcore.measurement.risk import scenario_cvar
|
||||||
|
from quantcore.optimization.problem import MeanCVaRProblem, OptimizationConstraints, ProblemSpec, TransactionCosts
|
||||||
|
from quantcore.optimization.solvers import HighsSolver
|
||||||
|
|
||||||
|
SETTINGS = Settings(cuopt_url="http://127.0.0.1:9")
|
||||||
|
|
||||||
|
|
||||||
|
def make_spec(seed=0, n=6, S=300, **kw):
|
||||||
|
rng = np.random.default_rng(seed)
|
||||||
|
symbols = [f"S{i}" for i in range(n)]
|
||||||
|
prices = rng.uniform(20, 400, n)
|
||||||
|
scen = rng.standard_normal((S, n)) * 0.03 + rng.uniform(-0.002, 0.006, n)
|
||||||
|
sectors = ["Tech", "Fin", "Health"] * (n // 3) + ["Tech"] * (n % 3)
|
||||||
|
base = dict(symbols=symbols, prices=prices, scenarios=scen, sectors=sectors, cash=100_000.0, confidence=0.95,
|
||||||
|
cvar_limit=4_000.0)
|
||||||
|
base.update(kw)
|
||||||
|
return ProblemSpec(**base)
|
||||||
|
|
||||||
|
|
||||||
|
def solve(spec):
|
||||||
|
problem = MeanCVaRProblem(spec)
|
||||||
|
model = problem.build()
|
||||||
|
res = HighsSolver(SETTINGS).solve(model, time_limit=60)
|
||||||
|
assert res.ok, res
|
||||||
|
return problem, model, res, problem.decode(res.x)
|
||||||
|
|
||||||
|
|
||||||
|
def test_new_portfolio_respects_all_constraints():
|
||||||
|
# with a 4000 limit the 10 % caps bind first (CVaR ~1040); at 800 the CVaR limit must bind
|
||||||
|
spec = make_spec(cvar_limit=800.0, constraints=OptimizationConstraints(max_weight=0.10, max_sector_weight=0.25))
|
||||||
|
problem, model, res, d = solve(spec)
|
||||||
|
W = spec.wealth
|
||||||
|
assert (d["shares"] >= 0).all() and np.issubdtype(d["shares"].dtype, np.integer)
|
||||||
|
assert d["cash_end"] >= -1e-6
|
||||||
|
assert (d["weights"] <= 0.10 + 1e-9).all()
|
||||||
|
for sec in set(spec.sectors):
|
||||||
|
assert d["value"][np.array(spec.sectors) == sec].sum() <= 0.25 * W + 1e-6
|
||||||
|
# the CVaR of the decoded portfolio (recomputed from the scenarios) is within the limit and binding
|
||||||
|
# up to the granularity of whole shares; alpha is *a* minimiser of the RU function, not necessarily the VaR
|
||||||
|
assert d["cvar"] <= spec.cvar_limit + 1e-6
|
||||||
|
assert d["cvar"] >= 0.98 * spec.cvar_limit
|
||||||
|
assert d["var"] <= d["cvar"]
|
||||||
|
# objective equals expected P&L net of costs
|
||||||
|
assert res.objective == pytest.approx(d["expected_pnl"], rel=1e-6)
|
||||||
|
assert d["total_cost"] == pytest.approx(0.001 * d["invested"], rel=1e-9)
|
||||||
|
|
||||||
|
|
||||||
|
def test_brute_force_optimality_two_assets():
|
||||||
|
spec = make_spec(n=2, S=80, cash=5_000.0, cvar_limit=300.0,
|
||||||
|
constraints=OptimizationConstraints(max_weight=0.6, max_sector_weight=1.0))
|
||||||
|
_, _, res, d = solve(spec)
|
||||||
|
p, scen = spec.prices, spec.scenarios
|
||||||
|
best, best_obj = None, -np.inf
|
||||||
|
max_n = (0.6 * spec.wealth / p).astype(int) + 1
|
||||||
|
for n0, n1 in itertools.product(range(max_n[0] + 1), range(max_n[1] + 1)):
|
||||||
|
n = np.array([n0, n1])
|
||||||
|
cost = 0.001 * (n * p).sum()
|
||||||
|
if (n * p).sum() + cost > spec.cash + 1e-9 or (n * p > 0.6 * spec.wealth + 1e-9).any():
|
||||||
|
continue
|
||||||
|
pnl = scen @ (n * p) - cost
|
||||||
|
_, cvar = scenario_cvar(-pnl, 0.95)
|
||||||
|
if cvar <= 300.0 + 1e-6 and pnl.mean() > best_obj:
|
||||||
|
best, best_obj = n, pnl.mean()
|
||||||
|
assert best_obj == pytest.approx(res.objective, rel=1e-6)
|
||||||
|
assert (d["shares"] == best).all()
|
||||||
|
|
||||||
|
|
||||||
|
def test_rebalancing_links_trades_and_costs():
|
||||||
|
holdings = np.array([50, 0, 20, 0, 10, 0], dtype=float)
|
||||||
|
spec = make_spec(cash=20_000.0, holdings=holdings, cvar_limit=3_000.0,
|
||||||
|
costs=TransactionCosts(0.002, 0.003, fixed_per_trade=25.0),
|
||||||
|
constraints=OptimizationConstraints(max_weight=0.15, max_sector_weight=0.4, max_turnover=0.5))
|
||||||
|
problem, model, res, d = solve(spec)
|
||||||
|
assert problem.rebalancing and problem.use_trade_bin
|
||||||
|
np.testing.assert_array_equal(d["shares"], holdings.astype(int) + d["buys"] - d["sells"])
|
||||||
|
assert not ((d["buys"] > 0) & (d["sells"] > 0)).any() # never buy and sell the same stock
|
||||||
|
n_trades = int(((d["buys"] + d["sells"]) > 0).sum())
|
||||||
|
expected_cost = 0.002 * (d["buys"] * spec.prices).sum() + 0.003 * (d["sells"] * spec.prices).sum() + 25.0 * n_trades
|
||||||
|
assert d["total_cost"] == pytest.approx(expected_cost)
|
||||||
|
assert d["cash_end"] == pytest.approx(spec.cash + (d["sells"] * spec.prices).sum() - (d["buys"] * spec.prices).sum() - d["total_cost"])
|
||||||
|
assert d["cash_end"] >= -1e-6
|
||||||
|
assert d["turnover"] <= 0.5 + 1e-9
|
||||||
|
assert d["cvar"] <= 3_000.0 + 1e-6
|
||||||
|
|
||||||
|
|
||||||
|
def test_objectives_min_cvar_and_utility():
|
||||||
|
spec = make_spec(objective="min_cvar", cvar_limit=None, target_pnl=40.0)
|
||||||
|
_, _, res, d = solve(spec)
|
||||||
|
assert d["expected_pnl"] >= 40.0 - 1e-6
|
||||||
|
assert res.objective == pytest.approx(d["cvar"], rel=1e-6)
|
||||||
|
spec_u = make_spec(objective="utility", cvar_limit=None, risk_aversion=0.5)
|
||||||
|
_, _, res_u, d_u = solve(spec_u)
|
||||||
|
assert res_u.objective == pytest.approx(d_u["expected_pnl"] - 0.5 * d_u["cvar"], rel=1e-6)
|
||||||
|
|
||||||
|
|
||||||
|
def test_cardinality_and_min_position():
|
||||||
|
spec = make_spec(cvar_limit=5_000.0, constraints=OptimizationConstraints(
|
||||||
|
max_weight=0.3, max_sector_weight=1.0, max_positions=2, min_position_value=5_000.0))
|
||||||
|
problem, model, res, d = solve(spec)
|
||||||
|
assert problem.use_pos_bin
|
||||||
|
held = d["value"] > 0
|
||||||
|
assert held.sum() <= 2
|
||||||
|
assert (d["value"][held] >= 5_000.0 - 1e-6).all()
|
||||||
|
|
||||||
|
|
||||||
|
def test_spec_validation():
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
make_spec(cvar_limit=None) # max_return needs a limit
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
make_spec(holdings=np.array([-1, 0, 0, 0, 0, 0]))
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
make_spec(objective="min_cvar", cvar_limit=None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_position_above_cap_is_sold_down_and_invested_bound():
|
||||||
|
holdings = np.zeros(6)
|
||||||
|
holdings[0] = 1000.0 # far above the 10 % cap
|
||||||
|
spec = make_spec(cash=10_000.0, holdings=holdings, cvar_limit=50_000.0,
|
||||||
|
constraints=OptimizationConstraints(max_weight=0.10, max_sector_weight=1.0))
|
||||||
|
_, model, res, d = solve(spec)
|
||||||
|
assert d["weights"][0] <= 0.10 + 1e-9 and d["sells"][0] > 0
|
||||||
|
# the single-stock bound is min(max_weight, max_invested) * W, not their product
|
||||||
|
spec2 = make_spec(cvar_limit=50_000.0, constraints=OptimizationConstraints(max_weight=0.10, max_sector_weight=1.0,
|
||||||
|
max_invested=0.5))
|
||||||
|
problem = MeanCVaRProblem(spec2)
|
||||||
|
np.testing.assert_array_equal(problem._max_shares(), np.floor(0.10 * spec2.wealth / spec2.prices + 1e-9))
|
||||||
|
spec3 = make_spec(cvar_limit=50_000.0, constraints=OptimizationConstraints(max_weight=0.6, max_sector_weight=1.0,
|
||||||
|
max_invested=0.5))
|
||||||
|
np.testing.assert_array_equal(MeanCVaRProblem(spec3)._max_shares(), np.floor(0.5 * spec3.wealth / spec3.prices + 1e-9))
|
||||||
|
|
||||||
|
|
||||||
|
def test_documentation_example_rounding_fails():
|
||||||
|
"""The two-stock instance of docs/optimization_model.tex, Example 4.1 (numbers as printed there)."""
|
||||||
|
from scipy.optimize import Bounds, LinearConstraint, milp
|
||||||
|
|
||||||
|
prices = np.array([572.0, 2208.0])
|
||||||
|
scen = np.array([[0.005, -0.082], [0.014, -0.052], [-0.069, -0.020], [-0.002, 0.055],
|
||||||
|
[-0.009, 0.083], [0.048, 0.064], [0.044, 0.028], [-0.011, 0.042]])
|
||||||
|
spec = ProblemSpec(symbols=["A", "B"], prices=prices, scenarios=scen, sectors=["X", "Y"], cash=10_000.0,
|
||||||
|
confidence=0.75, cvar_limit=250.0, costs=TransactionCosts(0.0, 0.0),
|
||||||
|
constraints=OptimizationConstraints(max_weight=0.7, max_sector_weight=1.0))
|
||||||
|
problem = MeanCVaRProblem(spec)
|
||||||
|
model = problem.build()
|
||||||
|
# LP relaxation: (2.6605, 1.7876), objective 62.02, CVaR binding
|
||||||
|
lp = milp(-model.c, constraints=LinearConstraint(model.A, model.row_lb, model.row_ub),
|
||||||
|
integrality=np.zeros(model.n_vars), bounds=Bounds(model.var_lb, model.var_ub))
|
||||||
|
np.testing.assert_allclose(lp.x[:2], [2.6605, 1.7876], atol=1e-4)
|
||||||
|
assert -lp.fun == pytest.approx(62.02, abs=0.01)
|
||||||
|
|
||||||
|
def evaluate(n):
|
||||||
|
pnl = scen @ (n * prices)
|
||||||
|
return pnl.mean(), scenario_cvar(-pnl, 0.75)[1]
|
||||||
|
|
||||||
|
assert evaluate(np.array([3, 2]))[1] == pytest.approx(280.13, abs=0.01) # nearest rounding: infeasible
|
||||||
|
assert evaluate(np.array([2, 1]))[0] == pytest.approx(35.43, abs=0.01) # rounding down: feasible, poor
|
||||||
|
res = HighsSolver(SETTINGS).solve(model)
|
||||||
|
assert res.ok
|
||||||
|
d = problem.decode(res.x)
|
||||||
|
np.testing.assert_array_equal(d["shares"], [7, 1]) # the integer optimum has a different structure
|
||||||
|
assert d["expected_pnl"] == pytest.approx(42.58, abs=0.01) and d["cvar"] == pytest.approx(240.74, abs=0.01)
|
||||||
67
tests/test_returns.py
Normal file
67
tests/test_returns.py
Normal file
@ -0,0 +1,67 @@
|
|||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from quantcore.measurement.returns import (
|
||||||
|
align_prices, horizon_periods, horizon_returns, log_returns, periods_per_year, portfolio_log_returns, resample_prices,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_log_returns_reference():
|
||||||
|
p = pd.DataFrame({"A": [100.0, 110.0, 99.0]}, index=pd.bdate_range("2024-01-01", periods=3))
|
||||||
|
r = log_returns(p)
|
||||||
|
assert len(r) == 2
|
||||||
|
assert r["A"].iloc[0] == pytest.approx(np.log(1.1))
|
||||||
|
assert r["A"].iloc[1] == pytest.approx(np.log(0.9))
|
||||||
|
|
||||||
|
|
||||||
|
def test_horizon_periods():
|
||||||
|
assert horizon_periods("day", "daily") == 1
|
||||||
|
assert horizon_periods("week", "daily") == 5
|
||||||
|
assert horizon_periods("month", "daily") == 21
|
||||||
|
assert horizon_periods("week", "weekly") == 1
|
||||||
|
assert horizon_periods("month", "weekly") == 4
|
||||||
|
assert horizon_periods(10, "daily") == 10
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
horizon_periods("day", "weekly")
|
||||||
|
assert periods_per_year("weekly") == 52
|
||||||
|
|
||||||
|
|
||||||
|
def test_align_and_resample():
|
||||||
|
idx = pd.bdate_range("2024-01-01", periods=10)
|
||||||
|
p = pd.DataFrame({"A": np.arange(10.0) + 1, "B": np.arange(10.0) + 1}, index=idx)
|
||||||
|
p.loc[idx[3], "B"] = np.nan # holiday on B's exchange
|
||||||
|
a = align_prices(p)
|
||||||
|
assert a.loc[idx[3], "B"] == a.loc[idx[2], "B"] # forward-filled
|
||||||
|
w = resample_prices(a, "weekly")
|
||||||
|
assert all(d.dayofweek == 4 for d in w.index)
|
||||||
|
assert w["A"].iloc[0] == 5.0 # Friday of the first week
|
||||||
|
|
||||||
|
|
||||||
|
def test_horizon_returns_sum():
|
||||||
|
r = pd.Series(np.arange(1.0, 11.0) / 100, index=pd.bdate_range("2024-01-01", periods=10))
|
||||||
|
h = horizon_returns(r, 5, overlapping=True)
|
||||||
|
assert len(h) == 6 and h.iloc[0] == pytest.approx(0.15)
|
||||||
|
nb = horizon_returns(r, 5, overlapping=False)
|
||||||
|
assert len(nb) == 2 and nb.iloc[-1] == pytest.approx(0.40)
|
||||||
|
|
||||||
|
|
||||||
|
def test_portfolio_log_returns_exact():
|
||||||
|
r = pd.DataFrame({"A": [np.log(1.1), np.log(0.9)], "B": [np.log(1.0), np.log(1.2)]})
|
||||||
|
rp = portfolio_log_returns(r, np.array([0.5, 0.5]))
|
||||||
|
assert rp.iloc[0] == pytest.approx(np.log(1 + 0.5 * 0.1 + 0.5 * 0.0))
|
||||||
|
assert rp.iloc[1] == pytest.approx(np.log(1 + 0.5 * (-0.1) + 0.5 * 0.2))
|
||||||
|
|
||||||
|
|
||||||
|
def test_monthly_resampling_works_on_both_pandas_spellings():
|
||||||
|
"""pandas renamed the month-end alias 'M' -> 'ME' in 2.2 and dropped 'M' in 3.0; the Mac runs
|
||||||
|
2.1 and the container 3.x, so the alias is resolved at import time."""
|
||||||
|
import pandas as pd
|
||||||
|
from quantcore.measurement.returns import MONTH_END, resample_prices
|
||||||
|
|
||||||
|
assert MONTH_END in ("M", "ME")
|
||||||
|
pd.tseries.frequencies.to_offset(MONTH_END) # accepted by the installed pandas
|
||||||
|
px = pd.DataFrame({"A": np.arange(400.0)}, index=pd.bdate_range("2024-01-01", periods=400))
|
||||||
|
monthly = resample_prices(px, "monthly")
|
||||||
|
assert len(monthly) == 19 and monthly.index.is_month_end.all()
|
||||||
|
assert (monthly["A"].diff().dropna() > 0).all() # month-end closes, in order
|
||||||
208
tests/test_risk.py
Normal file
208
tests/test_risk.py
Normal file
@ -0,0 +1,208 @@
|
|||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
from scipy import stats
|
||||||
|
|
||||||
|
from quantcore.measurement import risk as rk
|
||||||
|
|
||||||
|
|
||||||
|
def test_normal_var_cvar_reference():
|
||||||
|
# sigma = 1 %, mu = 0: VaR95 = 1.6449 %, CVaR95 = 2.0627 %, VaR99 = 2.3263 %, CVaR99 = 2.6652 %
|
||||||
|
assert rk.normal_var(0.0, 0.01, 0.95) == pytest.approx(0.016449, abs=1e-6)
|
||||||
|
assert rk.normal_cvar(0.0, 0.01, 0.95) == pytest.approx(0.020627, abs=1e-6)
|
||||||
|
assert rk.normal_var(0.0, 0.01, 0.99) == pytest.approx(0.023263, abs=1e-6)
|
||||||
|
assert rk.normal_cvar(0.0, 0.01, 0.99) == pytest.approx(0.026652, abs=1e-6)
|
||||||
|
# a positive mean lowers the loss quantile one-for-one
|
||||||
|
assert rk.normal_var(0.001, 0.01, 0.95) == pytest.approx(0.015449, abs=1e-6)
|
||||||
|
|
||||||
|
|
||||||
|
def test_historical_var_cvar_reference():
|
||||||
|
r = pd.Series(np.arange(-0.10, 0.10, 0.001)) # 200 equally spaced returns
|
||||||
|
var95 = rk.historical_var(r, 0.95)
|
||||||
|
assert var95 == pytest.approx(-np.quantile(r, 0.05))
|
||||||
|
cvar95 = rk.historical_cvar(r, 0.95)
|
||||||
|
tail = r[r <= np.quantile(r, 0.05)]
|
||||||
|
assert cvar95 == pytest.approx(-tail.mean())
|
||||||
|
assert cvar95 > var95
|
||||||
|
|
||||||
|
|
||||||
|
def test_scenario_cvar_is_tail_mean():
|
||||||
|
losses = np.array([1.0, 5.0, 3.0, 9.0, 7.0, 2.0, 8.0, 4.0, 6.0, 10.0]) # S = 10
|
||||||
|
var, cvar = rk.scenario_cvar(losses, 0.80) # k = 2 -> mean of the two largest
|
||||||
|
assert cvar == pytest.approx((10 + 9) / 2)
|
||||||
|
assert var == pytest.approx(9.0)
|
||||||
|
var, cvar = rk.scenario_cvar(losses, 0.95) # k = 0.5 -> fractional tail: 0.5 * 10 / 0.5
|
||||||
|
assert cvar == pytest.approx(10.0)
|
||||||
|
var, cvar = rk.scenario_cvar(losses, 0.85) # k = 1.5 -> (10 + 0.5 * 9) / 1.5
|
||||||
|
assert cvar == pytest.approx((10 + 0.5 * 9) / 1.5)
|
||||||
|
assert var == pytest.approx(9.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_scenario_cvar_matches_ru_minimisation():
|
||||||
|
rng = np.random.default_rng(0)
|
||||||
|
losses = rng.standard_normal(500)
|
||||||
|
beta = 0.95
|
||||||
|
_, cvar = rk.scenario_cvar(losses, beta)
|
||||||
|
grid = np.linspace(-1, 4, 20001)
|
||||||
|
ru = np.min([a + np.maximum(losses - a, 0).mean() / (1 - beta) for a in grid])
|
||||||
|
assert cvar == pytest.approx(ru, abs=1e-3)
|
||||||
|
|
||||||
|
|
||||||
|
def test_loss_to_money():
|
||||||
|
assert rk.loss_to_money(0.0, 1000.0) == 0.0
|
||||||
|
assert rk.loss_to_money(np.log(1 / 0.9), 1000.0) == pytest.approx(100.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_asset_risk_table_columns_and_scaling():
|
||||||
|
rng = np.random.default_rng(1)
|
||||||
|
r = pd.DataFrame(rng.standard_normal((1500, 2)) * 0.01, index=pd.bdate_range("2019-01-01", periods=1500), columns=["A", "B"])
|
||||||
|
day = rk.asset_risk_table(r, "daily", 200, 0.94, (0.95,), "day")
|
||||||
|
week = rk.asset_risk_table(r, "daily", 200, 0.94, (0.95,), "week")
|
||||||
|
assert {"vol_rolling_ann", "vol_ewm_ann", "var95_normal_rolling", "cvar95_hist"} <= set(day.columns)
|
||||||
|
assert week["horizon_periods"].iloc[0] == 5
|
||||||
|
# sqrt(5) scaling of the Gaussian VaR for a zero-mean window (approximately, mean is small)
|
||||||
|
ratio = week["var95_normal_rolling"] / day["var95_normal_rolling"]
|
||||||
|
assert ratio.between(2.0, 2.5).all()
|
||||||
|
assert (day["cvar95_normal_ewm"] > day["var95_normal_ewm"]).all()
|
||||||
|
|
||||||
|
|
||||||
|
def test_portfolio_risk_normal_vs_scenarios(md):
|
||||||
|
svc = rk.RiskService(md)
|
||||||
|
holdings = {"S0": 100, "S1": 50, "S2": 200}
|
||||||
|
pr = svc.portfolio_risk(holdings, cash=1000.0, horizon="week", method="normal")
|
||||||
|
assert pr.value > 0 and set(pr.confidence) == {"0.95", "0.99"}
|
||||||
|
assert pr.confidence["0.99"]["cvar_money"] > pr.confidence["0.95"]["cvar_money"] > 0
|
||||||
|
# historical estimate is of the same order of magnitude
|
||||||
|
ph = svc.portfolio_risk(holdings, cash=1000.0, horizon="week", method="historical")
|
||||||
|
assert 0.3 < ph.confidence["0.95"]["cvar_money"] / pr.confidence["0.95"]["cvar_money"] < 3.0
|
||||||
|
# scenarios: Gaussian scenarios with the same covariance reproduce the normal CVaR
|
||||||
|
rng = np.random.default_rng(0)
|
||||||
|
r = md.returns(list(holdings), currency="base", freq="daily")
|
||||||
|
from quantcore.measurement import estimators as est
|
||||||
|
cov = est.rolling_cov(r, 200)
|
||||||
|
mu_h = r.iloc[-200:].mean().values * 5
|
||||||
|
scen = rng.multivariate_normal(mu_h, cov.values * 5, size=200_000)
|
||||||
|
ps = svc.portfolio_risk(holdings, cash=1000.0, horizon="week", method="scenarios", scenarios=scen)
|
||||||
|
# differences: sampling noise and the exact money mapping (exp) vs. the log-return closed form
|
||||||
|
assert ps.confidence["0.95"]["cvar_money"] == pytest.approx(pr.confidence["0.95"]["cvar_money"], rel=0.06)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_one_asset_portfolio_is_a_valid_request(md):
|
||||||
|
"""``np.cov`` of a single column returns a 0-d array; the rolling estimator has to keep it 2-d.
|
||||||
|
|
||||||
|
The simplest call anyone makes -- the risk of a single position -- went through this path.
|
||||||
|
"""
|
||||||
|
svc = rk.RiskService(md)
|
||||||
|
from quantcore.measurement import estimators as est
|
||||||
|
|
||||||
|
r = md.returns(["S0"], currency="base", freq="daily")
|
||||||
|
assert est.rolling_cov(r, 200).shape == (1, 1)
|
||||||
|
assert est.ewm_cov(r, 0.94).shape == (1, 1)
|
||||||
|
for method in ("normal", "historical"):
|
||||||
|
for cov_method in ("rolling", "ewm"):
|
||||||
|
pr = svc.portfolio_risk({"S0": 100}, cash=1000.0, horizon="week", method=method,
|
||||||
|
cov_method=cov_method)
|
||||||
|
assert pr.value > 0 and pr.confidence["0.95"]["cvar_money"] > 0
|
||||||
|
# a one-asset portfolio is the asset: the parametric VaR is z * sd_h - mean_h exactly
|
||||||
|
one = svc.portfolio_risk({"S0": 100}, cash=0.0, horizon="week", method="normal", cov_method="rolling")
|
||||||
|
sd_week = float(np.sqrt(est.rolling_cov(r, 200).iloc[0, 0] * 5))
|
||||||
|
mean_week = float(r.iloc[-200:].mean().iloc[0] * 5)
|
||||||
|
assert one.confidence["0.95"]["var"] == pytest.approx(1.6448536269514722 * sd_week - mean_week, rel=1e-9)
|
||||||
|
|
||||||
|
|
||||||
|
def test_weekly_reference_values_from_documentation():
|
||||||
|
"""docs/measurement_and_models.tex, Example 4.1: sigma = 1 % daily, one week, V0 = 100 000."""
|
||||||
|
s_h = 0.01 * np.sqrt(5)
|
||||||
|
assert rk.normal_var(0.0, s_h, 0.95) == pytest.approx(0.036780, abs=1e-6)
|
||||||
|
assert rk.normal_cvar(0.0, s_h, 0.95) == pytest.approx(0.046124, abs=1e-6)
|
||||||
|
assert rk.normal_var(0.0, s_h, 0.99) == pytest.approx(0.052019, abs=1e-6)
|
||||||
|
assert rk.normal_cvar(0.0, s_h, 0.99) == pytest.approx(0.059596, abs=1e-6)
|
||||||
|
v0 = 100_000.0
|
||||||
|
assert rk.loss_to_money(rk.normal_var(0.0, s_h, 0.95), v0) == pytest.approx(3611.2, abs=0.1)
|
||||||
|
assert rk.loss_to_money(rk.normal_cvar(0.0, s_h, 0.95), v0) == pytest.approx(4507.6, abs=0.1)
|
||||||
|
assert rk.loss_to_money(rk.normal_var(0.0, s_h, 0.99), v0) == pytest.approx(5068.9, abs=0.1)
|
||||||
|
assert rk.loss_to_money(rk.normal_cvar(0.0, s_h, 0.99), v0) == pytest.approx(5785.5, abs=0.1)
|
||||||
|
|
||||||
|
|
||||||
|
def test_weighted_quantile_matches_numpy_and_handles_weights():
|
||||||
|
rng = np.random.default_rng(3)
|
||||||
|
x = rng.standard_normal(5000)
|
||||||
|
qs = (0.01, 0.05, 0.5, 0.95)
|
||||||
|
np.testing.assert_allclose(rk.weighted_quantile(x, qs), np.quantile(x, qs), rtol=1e-12)
|
||||||
|
# equal weights reproduce the unweighted result (the weighted convention places the
|
||||||
|
# observations at the midpoints of their probability steps, so the tails differ by half a rank)
|
||||||
|
w = np.full(len(x), 1 / len(x))
|
||||||
|
np.testing.assert_allclose(rk.weighted_quantile(x, qs, w), np.quantile(x, qs), atol=0.01)
|
||||||
|
# column-wise on a matrix
|
||||||
|
m = rng.standard_normal((2000, 3))
|
||||||
|
np.testing.assert_allclose(rk.weighted_quantile(m, (0.1, 0.9)), np.quantile(m, (0.1, 0.9), axis=0), rtol=1e-12)
|
||||||
|
# a sample that over-represents the left tail, corrected by its weights, recovers the standard normal
|
||||||
|
tilted = np.concatenate([rng.standard_normal(4000), rng.standard_normal(4000) - 2.0])
|
||||||
|
p = np.concatenate([np.full(4000, 1.0), stats.norm.pdf(tilted[4000:]) / stats.norm.pdf(tilted[4000:] + 2.0)])
|
||||||
|
p = p / p.sum()
|
||||||
|
for q in (0.01, 0.05, 0.5):
|
||||||
|
assert rk.weighted_quantile(tilted, q, p)[0] == pytest.approx(stats.norm.ppf(q), abs=0.12)
|
||||||
|
|
||||||
|
|
||||||
|
def test_value_loss_maps_a_log_loss_to_a_share_of_the_position():
|
||||||
|
assert rk.value_loss(0.0) == pytest.approx(0.0)
|
||||||
|
assert rk.value_loss(0.30) == pytest.approx(1 - np.exp(-0.30)) # 25.9 %, not 30 %
|
||||||
|
assert rk.value_loss(5.0) < 1.0 # can never reach 100 %
|
||||||
|
np.testing.assert_allclose(rk.value_loss([0.01, 0.1]), [1 - np.exp(-0.01), 1 - np.exp(-0.1)])
|
||||||
|
# a money loss is the value-space loss times the position
|
||||||
|
assert rk.loss_to_money(0.30, 1_000.0) == pytest.approx(1_000.0 * rk.value_loss(0.30))
|
||||||
|
|
||||||
|
|
||||||
|
def test_value_space_var_transforms_exactly_but_cvar_needs_the_sigma_squared_term():
|
||||||
|
"""``1 - e^{-x}`` is monotone, so a VaR carries over unchanged; a CVaR is a tail *mean* and
|
||||||
|
the concavity costs a ``sigma^2 / 2`` term -- the same correction as the Ito drift."""
|
||||||
|
m, s, beta = 0.01, 0.30, 0.95
|
||||||
|
r = np.random.default_rng(1).normal(m, s, 2_000_000)
|
||||||
|
loss = 1 - np.exp(r)
|
||||||
|
q = np.quantile(loss, beta)
|
||||||
|
assert rk.normal_value_var(m, s, beta) == pytest.approx(q, abs=2e-3)
|
||||||
|
assert rk.normal_value_var(m, s, beta) == pytest.approx(rk.value_loss(rk.normal_var(m, s, beta)), rel=1e-12)
|
||||||
|
exact = rk.normal_value_cvar(m, s, beta)
|
||||||
|
assert exact == pytest.approx(loss[loss >= q].mean(), abs=2e-3)
|
||||||
|
naive = rk.value_loss(rk.normal_cvar(m, s, beta)) # what converting afterwards would give
|
||||||
|
assert naive > exact # concavity: the naive figure overstates
|
||||||
|
assert 0.002 < naive - exact < 0.005
|
||||||
|
# the correction vanishes as sigma -> 0 and grows with sigma
|
||||||
|
assert rk.value_loss(rk.normal_cvar(m, 0.01, beta)) - rk.normal_value_cvar(m, 0.01, beta) < 1e-5
|
||||||
|
assert rk.value_loss(rk.normal_cvar(m, 0.6, beta)) - rk.normal_value_cvar(m, 0.6, beta) > 0.005
|
||||||
|
assert rk.normal_value_cvar(m, 3.0, beta) < 1.0 # never a total loss, whatever the volatility
|
||||||
|
|
||||||
|
|
||||||
|
def test_asset_risk_table_in_value_space():
|
||||||
|
rng = np.random.default_rng(5)
|
||||||
|
r = pd.DataFrame(rng.standard_normal((1500, 2)) * np.array([0.012, 0.03]),
|
||||||
|
index=pd.bdate_range("2019-01-01", periods=1500), columns=["A", "B"])
|
||||||
|
log = rk.asset_risk_table(r, "daily", horizon="month", confidence=(0.95,))
|
||||||
|
val = rk.asset_risk_table(r, "daily", horizon="month", confidence=(0.95,), space="value")
|
||||||
|
for col in ("var95_normal_ewm", "var95_hist"):
|
||||||
|
np.testing.assert_allclose(val[col], rk.value_loss(log[col]), rtol=1e-7) # quantiles: exact
|
||||||
|
for col in ("cvar95_normal_ewm", "cvar95_hist"):
|
||||||
|
assert (val[col] < rk.value_loss(log[col])).all() # tail means: corrected
|
||||||
|
assert (val[col] < 1.0).all() and (val[col] > 0).all()
|
||||||
|
np.testing.assert_allclose(val["vol_ewm_ann"], log["vol_ewm_ann"]) # only the losses move
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
rk.asset_risk_table(r, "daily", space="nonsense")
|
||||||
|
|
||||||
|
|
||||||
|
def test_zero_drift_removes_the_extrapolated_mean_from_the_parametric_risk():
|
||||||
|
"""At a quarterly horizon the estimated mean dominates the VaR: a stock with a strong recent
|
||||||
|
rally would otherwise look *less* risky than a quiet one."""
|
||||||
|
rng = np.random.default_rng(11)
|
||||||
|
r = pd.DataFrame({"trend": rng.standard_normal(1500) * 0.02 + 0.004, # strong drift
|
||||||
|
"quiet": rng.standard_normal(1500) * 0.02},
|
||||||
|
index=pd.bdate_range("2019-01-01", periods=1500))
|
||||||
|
est_ = rk.asset_risk_table(r, "daily", horizon="quarter", confidence=(0.95,))
|
||||||
|
zero = rk.asset_risk_table(r, "daily", horizon="quarter", confidence=(0.95,), drift="zero")
|
||||||
|
assert est_.loc["trend", "var95_normal_rolling"] < est_.loc["quiet", "var95_normal_rolling"]
|
||||||
|
assert zero.loc["trend", "var95_normal_rolling"] == pytest.approx(
|
||||||
|
zero.loc["quiet", "var95_normal_rolling"], rel=0.05) # same volatility -> same risk
|
||||||
|
assert zero.loc["trend", "var95_normal_rolling"] > est_.loc["trend", "var95_normal_rolling"]
|
||||||
|
# the historical columns keep the drift of their window either way
|
||||||
|
assert zero.loc["trend", "var95_hist"] == est_.loc["trend", "var95_hist"]
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
rk.asset_risk_table(r, "daily", drift="wishful")
|
||||||
275
tests/test_rowgen_gaussian.py
Normal file
275
tests/test_rowgen_gaussian.py
Normal file
@ -0,0 +1,275 @@
|
|||||||
|
"""Weighted scenarios, tail-row generation and the exact Gaussian (outer-approximation) mode."""
|
||||||
|
|
||||||
|
import itertools
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
from scipy.stats import norm
|
||||||
|
|
||||||
|
from quantcore.config import Settings
|
||||||
|
from quantcore.models import sampling as smp
|
||||||
|
from quantcore.optimization.gaussian_exact import solve_gaussian_exact
|
||||||
|
from quantcore.optimization.problem import (
|
||||||
|
MeanCVaRProblem,
|
||||||
|
OptimizationConstraints,
|
||||||
|
ProblemSpec,
|
||||||
|
TransactionCosts,
|
||||||
|
cvar_multiplier,
|
||||||
|
)
|
||||||
|
from quantcore.optimization.rowgen import restricted_spec, solve_by_row_generation
|
||||||
|
from quantcore.optimization.solvers import HighsSolver
|
||||||
|
|
||||||
|
SETTINGS = Settings(cuopt_url="http://127.0.0.1:9", mip_rel_gap=1e-6)
|
||||||
|
|
||||||
|
|
||||||
|
def highs(model):
|
||||||
|
return HighsSolver(SETTINGS).solve(model, time_limit=120)
|
||||||
|
|
||||||
|
|
||||||
|
def gaussian_inputs(seed=0, n=6):
|
||||||
|
rng = np.random.default_rng(seed)
|
||||||
|
symbols = [f"S{i}" for i in range(n)]
|
||||||
|
prices = rng.uniform(20, 400, n)
|
||||||
|
vol = rng.uniform(0.02, 0.06, n)
|
||||||
|
corr = 0.4 * np.ones((n, n)) + 0.6 * np.eye(n)
|
||||||
|
cov = corr * np.outer(vol, vol)
|
||||||
|
mean = rng.uniform(0.0, 0.006, n)
|
||||||
|
sectors = ["Tech", "Fin", "Health"] * (n // 3) + ["Tech"] * (n % 3)
|
||||||
|
return symbols, prices, mean, cov, sectors
|
||||||
|
|
||||||
|
|
||||||
|
def gaussian_scenarios(mean, cov, S, seed=1):
|
||||||
|
z = smp.moment_match(smp.standard_normal_sample(S, len(mean), "sobol", seed))
|
||||||
|
return mean + z @ smp.pca_root(cov).T
|
||||||
|
|
||||||
|
|
||||||
|
def test_weighted_scenarios_equal_duplicated_scenarios():
|
||||||
|
symbols, prices, mean, cov, sectors = gaussian_inputs()
|
||||||
|
scen = gaussian_scenarios(mean, cov, 256)
|
||||||
|
dup = np.vstack([scen, scen[:64]]) # the first 64 scenarios counted twice
|
||||||
|
p = np.ones(256)
|
||||||
|
p[:64] = 2
|
||||||
|
p /= p.sum()
|
||||||
|
common = dict(symbols=symbols, prices=prices, sectors=sectors, cash=100_000.0, cvar_limit=3_000.0)
|
||||||
|
a = MeanCVaRProblem(ProblemSpec(scenarios=dup, **common))
|
||||||
|
b = MeanCVaRProblem(ProblemSpec(scenarios=scen, probabilities=p, **common))
|
||||||
|
ra, rb = highs(a.build()), highs(b.build())
|
||||||
|
assert ra.ok and rb.ok
|
||||||
|
da, db = a.decode(ra.x), b.decode(rb.x)
|
||||||
|
assert abs(ra.objective - rb.objective) < 1e-6 * abs(ra.objective)
|
||||||
|
np.testing.assert_array_equal(da["shares"], db["shares"])
|
||||||
|
assert abs(da["cvar"] - db["cvar"]) < 1e-6 and abs(da["expected_pnl"] - db["expected_pnl"]) < 1e-6
|
||||||
|
|
||||||
|
|
||||||
|
def test_expected_returns_drive_the_objective():
|
||||||
|
symbols, prices, mean, cov, sectors = gaussian_inputs()
|
||||||
|
scen = gaussian_scenarios(mean, cov, 256)
|
||||||
|
common = dict(symbols=symbols, prices=prices, scenarios=scen, sectors=sectors, cash=100_000.0, cvar_limit=3_000.0,
|
||||||
|
costs=TransactionCosts(proportional_buy=0.0))
|
||||||
|
shifted = mean + np.array([0.01, 0, 0, 0, 0, 0]) # the model says S0 earns 1 % more than the scenarios show
|
||||||
|
prob = MeanCVaRProblem(ProblemSpec(expected_returns=shifted, **common))
|
||||||
|
res = highs(prob.build())
|
||||||
|
d = prob.decode(res.x)
|
||||||
|
assert abs(d["expected_pnl"] - (shifted @ d["value"])) < 1e-6 # objective uses the model mean ...
|
||||||
|
assert d["expected_pnl"] > d["expected_pnl_scenarios"] + 0.005 * d["value"][0] - 1e-6 # ... not the sample mean
|
||||||
|
assert d["shares"][0] > 0
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
ProblemSpec(expected_returns=np.zeros(3), **common)
|
||||||
|
|
||||||
|
|
||||||
|
def test_row_generation_matches_full_milp():
|
||||||
|
symbols, prices, mean, cov, sectors = gaussian_inputs(seed=2)
|
||||||
|
scen = gaussian_scenarios(mean, cov, 2048)
|
||||||
|
spec = ProblemSpec(symbols=symbols, prices=prices, scenarios=scen, sectors=sectors, cash=100_000.0,
|
||||||
|
cvar_limit=3_000.0, constraints=OptimizationConstraints(max_weight=0.3, max_sector_weight=0.6))
|
||||||
|
full = MeanCVaRProblem(spec)
|
||||||
|
res_full = highs(full.build())
|
||||||
|
res_rg, problem_rg, info = solve_by_row_generation(spec, highs)
|
||||||
|
assert res_rg.ok and info.converged and info.iterations >= 1
|
||||||
|
assert max(info.working_set_sizes) < 0.5 * spec.n_scenarios # solved on a fraction of the rows
|
||||||
|
d_full, d_rg = full.decode(res_full.x), problem_rg.decode(res_rg.x)
|
||||||
|
assert abs(d_rg["expected_pnl"] - d_full["expected_pnl"]) <= 1e-5 * abs(d_full["expected_pnl"]) + 1e-6
|
||||||
|
assert d_rg["cvar"] <= 3_000.0 * (1 + 1e-7)
|
||||||
|
# the expanded solution vector is feasible for the full model: u_s >= L_s - alpha for every scenario
|
||||||
|
m = full.build()
|
||||||
|
Ax = m.A @ res_rg.x
|
||||||
|
assert (Ax >= m.row_lb - 1e-6).all() and (Ax <= m.row_ub + 1e-6).all()
|
||||||
|
# a restricted spec keeps the full-set probabilities and expected returns
|
||||||
|
sub = restricted_spec(spec, np.arange(10))
|
||||||
|
assert sub.n_scenarios == 10 and abs(sub.probabilities.sum() - 10 / 2048) < 1e-12
|
||||||
|
np.testing.assert_allclose(sub.expected_returns, spec.mean_returns)
|
||||||
|
|
||||||
|
|
||||||
|
def test_row_generation_rebalancing_mode():
|
||||||
|
symbols, prices, mean, cov, sectors = gaussian_inputs(seed=3)
|
||||||
|
scen = gaussian_scenarios(mean, cov, 1024)
|
||||||
|
holdings = np.array([40, 0, 10, 0, 5, 0], dtype=float)
|
||||||
|
spec = ProblemSpec(symbols=symbols, prices=prices, scenarios=scen, sectors=sectors, cash=20_000.0,
|
||||||
|
holdings=holdings, cvar_limit=2_500.0, constraints=OptimizationConstraints(max_weight=0.3, max_sector_weight=0.6))
|
||||||
|
res, problem, info = solve_by_row_generation(spec, highs)
|
||||||
|
assert res.ok and info.converged
|
||||||
|
d = problem.decode(res.x)
|
||||||
|
np.testing.assert_array_equal(d["shares"], holdings + d["buys"] - d["sells"])
|
||||||
|
assert d["cvar"] <= 2_500.0 * (1 + 1e-7) and d["cash_end"] >= -1e-6
|
||||||
|
|
||||||
|
|
||||||
|
def brute_force_gaussian(prices, mean, cov, sectors, cash, limit, beta, max_weight, costs):
|
||||||
|
"""Enumerate all integer share vectors within the caps (small instance)."""
|
||||||
|
ub = np.floor(max_weight * cash / prices).astype(int)
|
||||||
|
k = cvar_multiplier(beta)
|
||||||
|
best, best_n = -np.inf, None
|
||||||
|
for shares in itertools.product(*[range(u + 1) for u in ub]):
|
||||||
|
s = np.array(shares, dtype=float)
|
||||||
|
v = s * prices
|
||||||
|
tc = costs * v.sum()
|
||||||
|
if v.sum() + tc > cash + 1e-9:
|
||||||
|
continue
|
||||||
|
pnl = mean @ v - tc
|
||||||
|
cvar = -pnl + k * np.sqrt(v @ cov @ v)
|
||||||
|
if cvar <= limit + 1e-9 and pnl > best:
|
||||||
|
best, best_n = pnl, s
|
||||||
|
return best, best_n
|
||||||
|
|
||||||
|
|
||||||
|
def test_gaussian_exact_matches_brute_force():
|
||||||
|
prices = np.array([400.0, 650.0, 900.0])
|
||||||
|
vol = np.array([0.03, 0.05, 0.04])
|
||||||
|
corr = np.array([[1, 0.5, 0.2], [0.5, 1, 0.3], [0.2, 0.3, 1]])
|
||||||
|
cov = corr * np.outer(vol, vol)
|
||||||
|
mean = np.array([0.003, 0.006, 0.004])
|
||||||
|
sectors = ["A", "B", "C"]
|
||||||
|
cash, limit, beta = 10_000.0, 500.0, 0.95
|
||||||
|
costs = TransactionCosts(proportional_buy=0.002)
|
||||||
|
spec = ProblemSpec(symbols=list("XYZ"), prices=prices, scenarios=None, sectors=sectors, cash=cash, cvar_limit=limit,
|
||||||
|
confidence=beta, gaussian_mean=mean, gaussian_cov=cov, costs=costs,
|
||||||
|
constraints=OptimizationConstraints(max_weight=0.6, max_sector_weight=1.0))
|
||||||
|
res, problem, info = solve_gaussian_exact(spec, highs, tol=1e-9)
|
||||||
|
assert res.ok and info.converged and info.cuts >= 2
|
||||||
|
d = problem.decode(res.x)
|
||||||
|
best, best_n = brute_force_gaussian(prices, mean, cov, sectors, cash, limit, beta, 0.6, 0.002)
|
||||||
|
np.testing.assert_array_equal(d["shares"], best_n)
|
||||||
|
assert abs(d["expected_pnl"] - best) < 1e-6
|
||||||
|
assert d["cvar"] <= limit * (1 + 1e-9)
|
||||||
|
# closed-form risk figures of the decoded portfolio
|
||||||
|
v = d["value"]
|
||||||
|
sigma = np.sqrt(v @ cov @ v)
|
||||||
|
assert abs(d["sigma"] - sigma) < 1e-9
|
||||||
|
assert abs(d["var"] - (-(mean @ v - d["total_cost"]) + norm.ppf(beta) * sigma)) < 1e-9
|
||||||
|
assert res.reproducible and "outer approximation" in res.message
|
||||||
|
|
||||||
|
|
||||||
|
def test_gaussian_exact_other_objectives_and_validation():
|
||||||
|
symbols, prices, mean, cov, sectors = gaussian_inputs(seed=4)
|
||||||
|
base = dict(symbols=symbols, prices=prices, scenarios=None, sectors=sectors, cash=100_000.0,
|
||||||
|
gaussian_mean=mean, gaussian_cov=cov)
|
||||||
|
# min CVaR at a target expected P&L: converges, meets the target, epigraph equals the true CVaR
|
||||||
|
spec = ProblemSpec(objective="min_cvar", target_pnl=150.0, **base)
|
||||||
|
res, problem, info = solve_gaussian_exact(spec, highs)
|
||||||
|
d = problem.decode(res.x)
|
||||||
|
assert info.converged and d["expected_pnl"] >= 150.0 - 1e-6
|
||||||
|
assert abs(d["cvar"] - d["cvar_model"]) <= 1e-6 * max(d["cvar"], 1.0)
|
||||||
|
# utility objective
|
||||||
|
spec_u = ProblemSpec(objective="utility", risk_aversion=0.05, **base) # at 0.5 staying in cash is optimal
|
||||||
|
res_u, problem_u, info_u = solve_gaussian_exact(spec_u, highs)
|
||||||
|
du = problem_u.decode(res_u.x)
|
||||||
|
assert info_u.converged and du["invested"] > 0
|
||||||
|
# the CVaR-limited solution is on the mean-variance frontier: no feasible portfolio with the same
|
||||||
|
# expected P&L has a smaller sigma (checked on the continuous relaxation via the cut model)
|
||||||
|
spec_m = ProblemSpec(cvar_limit=3_000.0, **base)
|
||||||
|
res_m, problem_m, _ = solve_gaussian_exact(spec_m, highs)
|
||||||
|
dm = problem_m.decode(res_m.x)
|
||||||
|
assert dm["cvar"] <= 3_000.0 * (1 + 1e-6)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
ProblemSpec(symbols=symbols, prices=prices, scenarios=None, sectors=sectors, cash=1.0, cvar_limit=1.0)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
MeanCVaRProblem(spec_m).build() # no cuts yet
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
solve_by_row_generation(spec_m, highs)
|
||||||
|
|
||||||
|
|
||||||
|
def test_moment_matched_sobol_scenarios_reproduce_the_exact_optimum():
|
||||||
|
"""The sample-average approximation with Gaussian scenarios (used linearly) converges to the exact
|
||||||
|
mean-variance-equivalent optimum; plain Monte Carlo at the same size does not."""
|
||||||
|
symbols, prices, mean, cov, sectors = gaussian_inputs(seed=6, n=8)
|
||||||
|
common = dict(symbols=symbols, prices=prices, sectors=sectors, cash=1_000_000.0, cvar_limit=30_000.0,
|
||||||
|
costs=TransactionCosts(proportional_buy=0.0),
|
||||||
|
constraints=OptimizationConstraints(max_weight=0.25, max_sector_weight=0.6))
|
||||||
|
exact_spec = ProblemSpec(scenarios=None, gaussian_mean=mean, gaussian_cov=cov, **common)
|
||||||
|
res_e, prob_e, _info = solve_gaussian_exact(exact_spec, highs, tol=1e-8)
|
||||||
|
d_e = prob_e.decode(res_e.x)
|
||||||
|
k = cvar_multiplier(0.95)
|
||||||
|
|
||||||
|
def true_cvar(v):
|
||||||
|
return -(mean @ v) + k * np.sqrt(v @ cov @ v)
|
||||||
|
|
||||||
|
# moment-matched Sobol' scenarios, model mean in the objective
|
||||||
|
scen = gaussian_scenarios(mean, cov, 4096, seed=3)
|
||||||
|
prob_s = MeanCVaRProblem(ProblemSpec(scenarios=scen, expected_returns=mean, **common))
|
||||||
|
d_s = prob_s.decode(highs(prob_s.build()).x)
|
||||||
|
assert abs(d_s["expected_pnl"] - d_e["expected_pnl"]) <= 0.01 * d_e["expected_pnl"]
|
||||||
|
assert true_cvar(d_s["value"]) <= 30_000.0 * 1.02
|
||||||
|
# plain Monte Carlo of the same size with the sample mean in the objective (the old default)
|
||||||
|
rng = np.random.default_rng(3)
|
||||||
|
mc = mean + rng.standard_normal((4096, 8)) @ np.linalg.cholesky(cov).T
|
||||||
|
prob_m = MeanCVaRProblem(ProblemSpec(scenarios=mc, **common))
|
||||||
|
d_m = prob_m.decode(highs(prob_m.build()).x)
|
||||||
|
true_pnl_mc = mean @ d_m["value"]
|
||||||
|
assert abs(true_pnl_mc - d_e["expected_pnl"]) > abs(d_s["expected_pnl"] - d_e["expected_pnl"])
|
||||||
|
assert d_m["expected_pnl"] > true_pnl_mc # in-sample optimism of the plain MC solution
|
||||||
|
|
||||||
|
|
||||||
|
def test_row_generation_stops_at_the_time_budget():
|
||||||
|
"""The time limit is the budget of the whole request, not of each pass.
|
||||||
|
|
||||||
|
Every pass is a MILP of its own, so a deadline that has already passed must end the loop with
|
||||||
|
the incumbent (feasible, not proven optimal) instead of starting another solve.
|
||||||
|
"""
|
||||||
|
import time
|
||||||
|
|
||||||
|
symbols, prices, mean, cov, sectors = gaussian_inputs(seed=2)
|
||||||
|
scen = gaussian_scenarios(mean, cov, 2048)
|
||||||
|
spec = ProblemSpec(symbols=symbols, prices=prices, scenarios=scen, sectors=sectors, cash=100_000.0,
|
||||||
|
cvar_limit=3_000.0, constraints=OptimizationConstraints(max_weight=0.3, max_sector_weight=0.6))
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def counted(model):
|
||||||
|
calls.append(1)
|
||||||
|
return highs(model)
|
||||||
|
|
||||||
|
res, problem, info = solve_by_row_generation(spec, counted, deadline=time.monotonic() - 1.0)
|
||||||
|
assert len(calls) == 1 and info.iterations == 1 and not info.converged
|
||||||
|
assert res.status == "FeasibleFound" and "time limit" in res.message
|
||||||
|
d = problem.decode(res.x) # the incumbent is a real portfolio, measured on all scenarios
|
||||||
|
assert d["shares"].sum() > 0 and np.isfinite(d["cvar"])
|
||||||
|
# with a budget in the future the loop runs to convergence as before
|
||||||
|
res2, _, info2 = solve_by_row_generation(spec, highs, deadline=time.monotonic() + 600)
|
||||||
|
assert info2.converged and res2.status == "Optimal"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_solve_function_hands_on_the_remaining_budget():
|
||||||
|
from quantcore.optimization.service import OptimizationRequest, OptimizationService
|
||||||
|
|
||||||
|
svc = OptimizationService.__new__(OptimizationService) # no data needed for this
|
||||||
|
svc.settings = SETTINGS
|
||||||
|
req = OptimizationRequest(time_limit=30.0, solver="highs", cvar_limit=1.0)
|
||||||
|
limits = []
|
||||||
|
deadline = __import__("time").monotonic() + 30.0
|
||||||
|
|
||||||
|
class Recorder:
|
||||||
|
def __init__(self, settings):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def solve(self, model, time_limit=None):
|
||||||
|
limits.append(time_limit)
|
||||||
|
return None
|
||||||
|
|
||||||
|
import quantcore.optimization.service as service_module
|
||||||
|
original = service_module.HighsSolver
|
||||||
|
service_module.HighsSolver = Recorder
|
||||||
|
try:
|
||||||
|
fn = svc._solve_fn(req, prefer_local=True, deadline=deadline)
|
||||||
|
fn(None)
|
||||||
|
fn(None)
|
||||||
|
finally:
|
||||||
|
service_module.HighsSolver = original
|
||||||
|
assert 0 < limits[1] <= limits[0] <= 30.0 # each call gets what is left, never the full limit again
|
||||||
122
tests/test_sampling.py
Normal file
122
tests/test_sampling.py
Normal file
@ -0,0 +1,122 @@
|
|||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
from scipy.stats import norm
|
||||||
|
|
||||||
|
from quantcore.models import sampling as smp
|
||||||
|
|
||||||
|
|
||||||
|
def test_next_power_of_two():
|
||||||
|
assert [smp.next_power_of_two(n) for n in (1, 2, 3, 2000, 2048, 2049, 100_000)] == [1, 2, 4, 2048, 2048, 4096, 131072]
|
||||||
|
|
||||||
|
|
||||||
|
def test_standard_normal_sample_methods():
|
||||||
|
for method in smp.SAMPLING_METHODS:
|
||||||
|
z = smp.standard_normal_sample(1024, 5, method, seed=3)
|
||||||
|
assert z.shape == (1024, 5) and np.isfinite(z).all()
|
||||||
|
anti = smp.standard_normal_sample(1000, 3, "antithetic", seed=1)
|
||||||
|
np.testing.assert_allclose(anti[:500], -anti[500:], atol=1e-15) # exact zero mean by construction
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
smp.standard_normal_sample(1000, 3, "sobol") # not a power of two
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
smp.standard_normal_sample(1024, 3, "bogus")
|
||||||
|
# Sobol' points are deterministic for a seed and stratify the marginals far better than MC
|
||||||
|
a = smp.standard_normal_sample(2048, 4, "sobol", seed=7)
|
||||||
|
b = smp.standard_normal_sample(2048, 4, "sobol", seed=7)
|
||||||
|
np.testing.assert_array_equal(a, b)
|
||||||
|
lhs = smp.standard_normal_sample(4096, 2, "lhs", seed=7)
|
||||||
|
counts = np.histogram(norm.cdf(lhs[:, 0]), bins=64, range=(0, 1))[0]
|
||||||
|
assert counts.min() == counts.max() == 64 # exactly 64 points per stratum
|
||||||
|
|
||||||
|
|
||||||
|
def test_pca_root():
|
||||||
|
rng = np.random.default_rng(0)
|
||||||
|
L = rng.standard_normal((6, 6))
|
||||||
|
cov = L @ L.T
|
||||||
|
a = smp.pca_root(cov)
|
||||||
|
np.testing.assert_allclose(a @ a.T, cov, atol=1e-10)
|
||||||
|
vals, vecs = np.linalg.eigh(cov)
|
||||||
|
# first column = largest-variance direction, columns orthogonal
|
||||||
|
assert abs(abs(a[:, 0] @ vecs[:, -1]) - np.sqrt(vals[-1])) < 1e-9
|
||||||
|
np.testing.assert_allclose(a.T @ a, np.diag(np.sort(vals)[::-1]), atol=1e-9)
|
||||||
|
|
||||||
|
|
||||||
|
def test_moment_match_exact_and_weighted():
|
||||||
|
rng = np.random.default_rng(1)
|
||||||
|
z = rng.standard_normal((500, 7)) * 1.3 + 0.2
|
||||||
|
m = smp.moment_match(z)
|
||||||
|
np.testing.assert_allclose(m.mean(axis=0), 0.0, atol=1e-13)
|
||||||
|
np.testing.assert_allclose(m.T @ m / len(m), np.eye(7), atol=1e-12)
|
||||||
|
p = rng.uniform(0.5, 2.0, 500)
|
||||||
|
p /= p.sum()
|
||||||
|
mw = smp.moment_match(z, p, target_variance=5.0)
|
||||||
|
np.testing.assert_allclose(p @ mw, 0.0, atol=1e-12)
|
||||||
|
np.testing.assert_allclose((mw * p[:, None]).T @ mw, 5.0 * np.eye(7), atol=1e-11)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
smp.moment_match(rng.standard_normal((5, 7)))
|
||||||
|
|
||||||
|
|
||||||
|
def test_tilt_normal_sample_weights_bounded_and_tail_enriched():
|
||||||
|
rng = np.random.default_rng(2)
|
||||||
|
n, d = 4096, 8
|
||||||
|
z = smp.standard_normal_sample(n, d, "sobol", seed=5)
|
||||||
|
cov = np.eye(d) * 0.01
|
||||||
|
root = smp.pca_root(cov)
|
||||||
|
exposure = rng.uniform(1, 3, d) * 1000
|
||||||
|
direction = smp.loss_direction(root, exposure)
|
||||||
|
assert abs(np.linalg.norm(direction) - 1) < 1e-12
|
||||||
|
beta = 0.95
|
||||||
|
zt, p = smp.tilt_normal_sample(z, direction, norm.ppf(beta), mixture=0.5)
|
||||||
|
assert p.shape == (n,) and abs(p.sum() - 1) < 1e-12 and (p > 0).all()
|
||||||
|
assert p.max() <= 1.0 / (1 - 0.5) / n * 1.1 # likelihood ratios bounded by 1/(1-mixture), self-normalised
|
||||||
|
ess = smp.effective_sample_size(p)
|
||||||
|
assert 0.5 * n < ess < n
|
||||||
|
# the loss of the reference exposure exceeds its VaR far more often than 5 % of the time
|
||||||
|
loss_nominal = -(z @ root.T) @ exposure
|
||||||
|
loss_tilted = -(zt @ root.T) @ exposure
|
||||||
|
var = np.quantile(loss_nominal, beta)
|
||||||
|
assert (loss_tilted > var).mean() > 0.2
|
||||||
|
# ... while the weighted tail probability is still ~ 1 - beta (unbiased)
|
||||||
|
assert abs(p[loss_tilted > var].sum() - (1 - beta)) < 0.01
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
smp.tilt_normal_sample(z, direction, 1.0, mixture=1.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_tilted_day_probabilities_and_resampling():
|
||||||
|
rng = np.random.default_rng(3)
|
||||||
|
scores = rng.standard_t(5, 2000)
|
||||||
|
q, theta = smp.tilted_day_probabilities(scores, target_mean=0.7)
|
||||||
|
assert theta > 0 and abs(q @ scores - 0.7) < 1e-8 and abs(q.sum() - 1) < 1e-12
|
||||||
|
q0, theta0 = smp.tilted_day_probabilities(scores, target_mean=-1.0)
|
||||||
|
assert theta0 == 0.0 and np.allclose(q0, 1 / 2000)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
smp.tilted_day_probabilities(scores, target_mean=scores.max() + 1)
|
||||||
|
idx, p, _theta = smp.tilted_resample(scores, 1000, 5, 0.7, seed=1, mixture=0.5)
|
||||||
|
assert idx.shape == (1000, 5) and abs(p.sum() - 1) < 1e-12
|
||||||
|
assert p.max() <= 2.0 / 1000 * 1.1 # likelihood ratios <= 1/(1-mixture), self-normalised
|
||||||
|
# tilted half draws high scores more often
|
||||||
|
assert scores[idx[:500]].mean() > scores[idx[500:]].mean() + 0.3
|
||||||
|
|
||||||
|
|
||||||
|
def test_resample_indices_methods():
|
||||||
|
T, S, h = 300, 1024, 5
|
||||||
|
mc = smp.resample_indices(T, S, h, "mc", seed=0)
|
||||||
|
assert mc.shape == (S, h) and mc.min() >= 0 and mc.max() < T
|
||||||
|
bal = smp.resample_indices(T, S, h, "balanced", seed=0)
|
||||||
|
counts = np.bincount(bal.ravel(), minlength=T)
|
||||||
|
assert counts.max() - counts.min() <= 1 # every observation used equally often
|
||||||
|
order = np.argsort(np.random.default_rng(1).standard_normal(T))
|
||||||
|
sob = smp.resample_indices(T, S, h, "sobol", seed=0, order=order)
|
||||||
|
assert sob.shape == (S, h) and set(np.unique(sob)) <= set(range(T))
|
||||||
|
ranks = smp.resample_indices(T, S, h, "sobol", seed=0)
|
||||||
|
np.testing.assert_array_equal(order[ranks], sob)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
smp.resample_indices(T, 1000, h, "sobol")
|
||||||
|
|
||||||
|
|
||||||
|
def test_scenario_hash():
|
||||||
|
rng = np.random.default_rng(4)
|
||||||
|
x = rng.standard_normal((50, 3))
|
||||||
|
h = smp.scenario_hash(x)
|
||||||
|
assert len(h) == 16 and h == smp.scenario_hash(x.copy())
|
||||||
|
assert h != smp.scenario_hash(x + 1e-9)
|
||||||
|
assert h != smp.scenario_hash(x, np.full(50, 0.02))
|
||||||
281
tests/test_service_equivalence.py
Normal file
281
tests/test_service_equivalence.py
Normal file
@ -0,0 +1,281 @@
|
|||||||
|
"""Service-level checks: the scenario optimiser reproduces the exact Gaussian (mean-variance-equivalent)
|
||||||
|
optimum when the scenarios are moment-matched Sobol' points, the drift convention is arithmetic, and
|
||||||
|
importance sampling / row generation leave the optimum unchanged."""
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from quantcore.optimization.problem import MeanCVaRProblem
|
||||||
|
from quantcore.optimization.service import OptimizationRequest, OptimizationService
|
||||||
|
from tests.conftest import SYMBOLS
|
||||||
|
|
||||||
|
BASE = dict(symbols=SYMBOLS, cash=500_000.0, cvar_limit=12_000.0, solver="highs", horizon="week",
|
||||||
|
costs={"proportional_buy": 0.0, "proportional_sell": 0.0}, importance_sampling="off",
|
||||||
|
constraints={"max_weight": 0.15, "max_sector_weight": 0.4})
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def svc(settings, md):
|
||||||
|
return OptimizationService(settings, md)
|
||||||
|
|
||||||
|
|
||||||
|
def gaussian_cvar_of(svc, req, result):
|
||||||
|
"""Closed-form CVaR of a proposal under the Gaussian model of the request."""
|
||||||
|
symbols = svc.candidate_symbols(req)
|
||||||
|
mu_ann, _, _ = svc.expected_returns(req, symbols)
|
||||||
|
spec, _ = svc.build_gaussian_spec(req, symbols, mu_ann)
|
||||||
|
shares = {p["symbol"]: p["shares"] for p in result.positions}
|
||||||
|
value = np.array([shares.get(s, 0) for s in symbols]) * spec.prices
|
||||||
|
return MeanCVaRProblem(spec).gaussian_risk(value, result.total_cost)
|
||||||
|
|
||||||
|
|
||||||
|
def test_sobol_moment_matched_scenarios_reproduce_the_exact_optimum(svc):
|
||||||
|
exact = svc.optimize(OptimizationRequest(scenario_model="gaussian_exact", **BASE))
|
||||||
|
assert exact.status == "Optimal" and exact.reproducible and exact.solver_info["outer_approximation"]["converged"]
|
||||||
|
scen = svc.optimize(OptimizationRequest(scenario_model="gbm", n_scenarios=4096, **BASE))
|
||||||
|
assert scen.status == "Optimal" and scen.model["scenarios"]["moment_matched"]
|
||||||
|
# The GBM scenarios are lognormal in simple returns while the exact mode is normal: at weekly
|
||||||
|
# volatilities of 4-5 % the lognormal left tail is ~2-3 % thinner, so the scenario optimum takes
|
||||||
|
# ~3 % more exposure. Within that model difference the two agree (the sharp test of the sampling
|
||||||
|
# itself, with Gaussian scenarios used linearly, is in test_rowgen_gaussian.py).
|
||||||
|
assert abs(scen.expected_pnl - exact.expected_pnl) <= 0.05 * exact.expected_pnl
|
||||||
|
_, cvar_true, _ = gaussian_cvar_of(svc, OptimizationRequest(scenario_model="gaussian_exact", **BASE), scen)
|
||||||
|
assert cvar_true <= BASE["cvar_limit"] * 1.05
|
||||||
|
# the expected horizon return reported per position is the model's (arithmetic) expectation
|
||||||
|
mu_ann, _, _ = svc.expected_returns(OptimizationRequest(**BASE), SYMBOLS)
|
||||||
|
for p in scen.positions:
|
||||||
|
assert abs(p["expected_return_horizon"] - mu_ann[p["symbol"]] / 252 * 5) < 1e-12
|
||||||
|
# plain Monte Carlo without moment matching: the sample mean disagrees with the model mean
|
||||||
|
mc = svc.optimize(OptimizationRequest(scenario_model="gbm", n_scenarios=4096, sampling="mc", moment_match=False, **BASE))
|
||||||
|
assert mc.status == "Optimal"
|
||||||
|
dev_mc = abs(mc.solver_info["objective"] - mc.expected_pnl) # objective uses the model mean either way
|
||||||
|
assert dev_mc < 1e-6 # (the objective is always the model expectation) ...
|
||||||
|
scen_mean_mc = svc.build_spec(OptimizationRequest(scenario_model="gbm", n_scenarios=4096, sampling="mc",
|
||||||
|
moment_match=False, **BASE), SYMBOLS,
|
||||||
|
svc.scenarios(OptimizationRequest(scenario_model="gbm", n_scenarios=4096, sampling="mc",
|
||||||
|
moment_match=False, **BASE), SYMBOLS, mu_ann))
|
||||||
|
# ... but the sample mean of a plain MC set is visibly off the model mean, a moment-matched one is not
|
||||||
|
err_mc = np.abs(scen_mean_mc.scenarios.mean(axis=0) - scen_mean_mc.expected_returns).max()
|
||||||
|
sobol_spec = svc.build_spec(OptimizationRequest(scenario_model="gbm", n_scenarios=4096, **BASE), SYMBOLS,
|
||||||
|
svc.scenarios(OptimizationRequest(scenario_model="gbm", n_scenarios=4096, **BASE), SYMBOLS, mu_ann))
|
||||||
|
err_sobol = np.abs(sobol_spec.scenarios.mean(axis=0) - sobol_spec.expected_returns).max()
|
||||||
|
assert err_sobol < 0.1 * err_mc
|
||||||
|
|
||||||
|
|
||||||
|
def test_importance_sampling_and_row_generation_keep_the_optimum(svc):
|
||||||
|
plain = svc.optimize(OptimizationRequest(scenario_model="gbm", n_scenarios=4096, **BASE))
|
||||||
|
tilted = svc.optimize(OptimizationRequest(scenario_model="gbm", n_scenarios=4096,
|
||||||
|
**{**BASE, "importance_sampling": "market"}))
|
||||||
|
assert tilted.status == "Optimal" and tilted.model["scenarios"]["weighted"]
|
||||||
|
assert tilted.model["scenarios"]["effective_sample_size"] > 2000
|
||||||
|
assert abs(tilted.expected_pnl - plain.expected_pnl) <= 0.02 * plain.expected_pnl
|
||||||
|
exact_req = OptimizationRequest(scenario_model="gaussian_exact", **BASE)
|
||||||
|
assert gaussian_cvar_of(svc, exact_req, tilted)[1] <= BASE["cvar_limit"] * 1.05
|
||||||
|
rg = svc.optimize(OptimizationRequest(scenario_model="gbm", n_scenarios=4096, **{**BASE, "row_generation": "on"}))
|
||||||
|
assert rg.status == "Optimal" and rg.solver_info["row_generation"]["converged"]
|
||||||
|
assert rg.scenario_hash == plain.scenario_hash # same scenario set ...
|
||||||
|
assert abs(rg.expected_pnl - plain.expected_pnl) <= 1e-4 * plain.expected_pnl # ... same optimum
|
||||||
|
assert max(rg.solver_info["row_generation"]["working_set_sizes"]) < 4096
|
||||||
|
|
||||||
|
|
||||||
|
def test_empirical_model_options(svc):
|
||||||
|
req = OptimizationRequest(scenario_model="empirical", n_scenarios=2048, **{**BASE, "importance_sampling": "market"})
|
||||||
|
res = svc.optimize(req)
|
||||||
|
assert res.status == "Optimal" and res.model["scenarios"]["model"] == "empirical"
|
||||||
|
assert all(t > 0 for t in res.model["scenarios"]["importance_sampling"]["theta"])
|
||||||
|
assert res.model["scenarios"]["importance_sampling"]["direction"] == "market"
|
||||||
|
assert res.cvar <= BASE["cvar_limit"] * (1 + 1e-6)
|
||||||
|
bal = svc.optimize(OptimizationRequest(scenario_model="empirical", n_scenarios=1000, sampling="balanced", **BASE))
|
||||||
|
assert bal.status == "Optimal" and bal.model["scenarios"]["n_scenarios"] == 1000
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
OptimizationRequest(scenario_model="bogus", **BASE)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
OptimizationRequest(**{**BASE, "importance_sampling": "maybe"})
|
||||||
|
|
||||||
|
|
||||||
|
def test_one_scenario_set_for_a_family_of_runs(svc):
|
||||||
|
"""Risk level, constraints, costs, objective, cash and holdings must not change the scenarios;
|
||||||
|
a different expected-return model must change only their drift, not their shocks."""
|
||||||
|
svc.clear_scenario_cache()
|
||||||
|
base = dict(BASE, importance_sampling="market", scenario_model="gbm", n_scenarios=2048)
|
||||||
|
ref = svc.optimize(OptimizationRequest(**base))
|
||||||
|
variants = {
|
||||||
|
"other CVaR limit": dict(base, cvar_limit=6_000.0),
|
||||||
|
"limit as a fraction": dict(base, cvar_limit=None, cvar_limit_fraction=0.02),
|
||||||
|
"confidence 99 %": dict(base, confidence=0.99, cvar_limit=20_000.0),
|
||||||
|
"tighter caps": dict(base, constraints={"max_weight": 0.10, "max_sector_weight": 0.3}),
|
||||||
|
"with costs": dict(base, costs={"proportional_buy": 0.002, "proportional_sell": 0.002}),
|
||||||
|
"min-CVaR objective": dict(base, objective="min_cvar", cvar_limit=None, target_return=0.001),
|
||||||
|
"utility objective": dict(base, objective="utility", cvar_limit=None, risk_aversion=0.2),
|
||||||
|
"more cash": dict(base, cash=2_000_000.0),
|
||||||
|
"rebalancing": dict(base, mode="rebalance", cash=50_000.0,
|
||||||
|
holdings={p["symbol"]: p["shares"] for p in ref.positions[:3]}),
|
||||||
|
}
|
||||||
|
for label, kwargs in variants.items():
|
||||||
|
res = svc.optimize(OptimizationRequest(**kwargs))
|
||||||
|
assert res.status == "Optimal", label
|
||||||
|
assert res.scenario_key["id"] == ref.scenario_key["id"], label
|
||||||
|
assert res.scenario_hash == ref.scenario_hash, label
|
||||||
|
assert res.solver_info["scenario_cache"]["reused"], label # and it was not regenerated
|
||||||
|
# a different expected-return model shifts the scenarios but leaves the shocks alone
|
||||||
|
symbols = svc.candidate_symbols(OptimizationRequest(**base))
|
||||||
|
req = OptimizationRequest(**base)
|
||||||
|
mu_a, _, _ = svc.expected_returns(req, symbols)
|
||||||
|
scen_a = svc.scenarios(req, symbols, mu_a)
|
||||||
|
req_views = OptimizationRequest(**dict(base, views=[{"symbol": SYMBOLS[0], "sentiment": 0.9, "confidence": 0.8}]))
|
||||||
|
mu_b, _, _ = svc.expected_returns(req_views, symbols)
|
||||||
|
scen_b = svc.scenarios(req_views, symbols, mu_b)
|
||||||
|
assert scen_b.hash != scen_a.hash and scen_b.shock_hash == scen_a.shock_hash
|
||||||
|
np.testing.assert_allclose(scen_b.shocks, scen_a.shocks, atol=1e-15)
|
||||||
|
assert (scen_b.expected_simple[0] > scen_a.expected_simple[0]) # the view moved the drift
|
||||||
|
# what *must* change the set: symbols, horizon, covariance model, seed, sampling, scenario count
|
||||||
|
for label, kwargs in {
|
||||||
|
"fewer symbols": dict(base, symbols=SYMBOLS[:8]),
|
||||||
|
"monthly horizon": dict(base, horizon="month"),
|
||||||
|
"other covariance": dict(base, covariance={"method": "rolling", "window": 250}),
|
||||||
|
"other seed": dict(base, seed=7),
|
||||||
|
"plain Monte Carlo": dict(base, sampling="mc"),
|
||||||
|
"more scenarios": dict(base, n_scenarios=4096),
|
||||||
|
"no tilt": dict(base, importance_sampling="off"),
|
||||||
|
}.items():
|
||||||
|
other = OptimizationRequest(**kwargs)
|
||||||
|
assert svc.scenario_key(other, svc.candidate_symbols(other))["id"] != ref.scenario_key["id"], label
|
||||||
|
# the adaptive tilt is honest about depending on the risk parameters
|
||||||
|
ad = OptimizationRequest(**dict(base, importance_sampling="adaptive"))
|
||||||
|
ad_other = OptimizationRequest(**dict(base, importance_sampling="adaptive", cvar_limit=6_000.0))
|
||||||
|
assert svc.scenario_key(ad, symbols)["id"] != svc.scenario_key(ad_other, symbols)["id"]
|
||||||
|
assert "adaptive_tilt" in svc.scenario_key(ad, symbols)
|
||||||
|
|
||||||
|
|
||||||
|
def test_tilt_is_driven_by_the_covariance_and_reports_its_alignment(svc):
|
||||||
|
"""The tilt shifts each asset by its covariance with the reference portfolio (not by its own
|
||||||
|
volatility), leaves the model correlations exactly intact, and the result says how well the
|
||||||
|
solution's loss lines up with the tilted direction."""
|
||||||
|
from scipy.stats import norm
|
||||||
|
|
||||||
|
from quantcore.models import sampling as smp
|
||||||
|
from quantcore.models.covariance import CovarianceModel
|
||||||
|
from quantcore.models.simulation import GBMSimulator, TiltSpec
|
||||||
|
from quantcore.optimization.problem import OptimizationConstraints, ProblemSpec, TransactionCosts
|
||||||
|
|
||||||
|
req = OptimizationRequest(**dict(BASE, scenario_model="gbm", n_scenarios=2048, importance_sampling="market"))
|
||||||
|
symbols = svc.candidate_symbols(req)
|
||||||
|
r, cov_model = svc._returns_and_cov_model(req, symbols)
|
||||||
|
h = req.horizon_periods
|
||||||
|
cov_h = cov_model.estimate(r).values * h
|
||||||
|
root = smp.pca_root(cov_h)
|
||||||
|
reference = svc.market_exposure(symbols) * 1e6
|
||||||
|
|
||||||
|
# 1. the shift of a tilted block is exactly -z_beta * Sigma v / sigma: proportional to each
|
||||||
|
# asset's covariance with the reference, i.e. to its beta, not to its own volatility
|
||||||
|
z = norm.ppf(0.95)
|
||||||
|
direction = smp.loss_direction(root, reference)
|
||||||
|
shift = z * (root @ direction)
|
||||||
|
sigma = np.sqrt(reference @ cov_h @ reference)
|
||||||
|
np.testing.assert_allclose(shift, -z * (cov_h @ reference) / sigma, atol=1e-12)
|
||||||
|
beta_i = (cov_h @ reference) / (reference @ cov_h @ reference)
|
||||||
|
assert np.corrcoef(shift, beta_i)[0, 1] < -0.999 # perfectly ordered by beta
|
||||||
|
if np.std(np.sqrt(np.diag(cov_h))) > 0: # ... and only loosely by own volatility
|
||||||
|
assert np.corrcoef(shift, np.sqrt(np.diag(cov_h)))[0, 1] > -0.999
|
||||||
|
|
||||||
|
# 2. the weighted tilted sample has exactly the model covariance (and correlations)
|
||||||
|
mu_ann, _, _ = svc.expected_returns(req, symbols)
|
||||||
|
sim = GBMSimulator(cov_model)
|
||||||
|
tilted = sim.simulate(r, n_scenarios=2048, horizon_periods=h, mu=mu_ann / 252, seed=1,
|
||||||
|
tilt=TiltSpec(exposure=reference))
|
||||||
|
w = tilted.weights
|
||||||
|
centred = tilted.log_returns - w @ tilted.log_returns
|
||||||
|
cov_t = (centred * w[:, None]).T @ centred
|
||||||
|
np.testing.assert_allclose(cov_t, cov_h, rtol=1e-8, atol=1e-14)
|
||||||
|
unweighted = np.cov(tilted.log_returns, rowvar=False)
|
||||||
|
assert np.abs(unweighted - cov_h).max() > 1e-6 # the raw tilted sample is *not* the model
|
||||||
|
|
||||||
|
# 3. alignment: 1 for the reference itself, negative for a portfolio built to oppose it
|
||||||
|
spec = ProblemSpec(scenarios=tilted.simple_returns(), probabilities=tilted.probabilities,
|
||||||
|
expected_returns=tilted.expected_simple, **svc._spec_common(req, symbols))
|
||||||
|
assert abs(svc.tilt_alignment(spec, reference, svc.market_exposure(symbols)) - 1) < 1e-9 # itself
|
||||||
|
opposed = np.zeros(len(symbols))
|
||||||
|
opposed[np.argsort(beta_i)[:2]] = 1e5 # the two assets least exposed to the reference
|
||||||
|
assert svc.tilt_alignment(spec, opposed, svc.market_exposure(symbols)) < 0.9
|
||||||
|
# a real run reports the figure and stays well above the warning threshold
|
||||||
|
res = svc.optimize(req)
|
||||||
|
assert res.solver_info["tilt_alignment"] > 0.5 and not res.warnings
|
||||||
|
|
||||||
|
|
||||||
|
def test_risk_drift_zero_keeps_the_limit_binding_when_the_forecast_is_large(md, settings):
|
||||||
|
"""Over a long horizon a big expected return shifts the whole tail into profit, and a CVaR limit
|
||||||
|
measured on the drifted scenarios stops constraining anything. The zero-drift convention measures
|
||||||
|
risk on the centred scenarios -- so the risk figure does not depend on the return forecast at all
|
||||||
|
-- while the objective still uses it."""
|
||||||
|
svc = OptimizationService(settings, md)
|
||||||
|
base = dict(symbols=SYMBOLS[:8], cash=1_000_000.0, horizon=252, freq="daily", confidence=0.99,
|
||||||
|
cvar_limit_fraction=0.12, scenario_model="gbm", n_scenarios=2048, solver="highs",
|
||||||
|
expected_returns="zero", row_generation="off", importance_sampling="off")
|
||||||
|
symbols = svc.candidate_symbols(OptimizationRequest(**base))
|
||||||
|
v = np.full(len(symbols), 125_000.0) # an equally weighted portfolio of the whole wealth
|
||||||
|
|
||||||
|
risk = {}
|
||||||
|
for mu_val in (0.8, 1.5): # 80 % and 150 % p.a. forecasts
|
||||||
|
for mode in ("model", "zero"):
|
||||||
|
req = OptimizationRequest(**{**base, "risk_drift": mode})
|
||||||
|
spec = svc.build_spec(req, symbols, svc.scenarios(req, symbols, pd.Series(mu_val, index=symbols)))
|
||||||
|
risk[(mu_val, mode)] = OptimizationService.portfolio_risk_on_scenarios(spec, v / spec.prices, 0.99)
|
||||||
|
if mode == "zero":
|
||||||
|
np.testing.assert_allclose(spec.mean_returns, svc.build_spec(
|
||||||
|
OptimizationRequest(**{**base, "risk_drift": "model"}), symbols,
|
||||||
|
svc.scenarios(OptimizationRequest(**base), symbols, pd.Series(mu_val, index=symbols))
|
||||||
|
).mean_returns) # the objective sees the forecast either way
|
||||||
|
|
||||||
|
# the zero-drift risk is the same whatever the forecast; the drifted one is not
|
||||||
|
assert risk[(0.8, "zero")]["cvar"] == pytest.approx(risk[(1.5, "zero")]["cvar"], rel=1e-12)
|
||||||
|
assert risk[(1.5, "model")]["cvar"] < 0 < risk[(1.5, "zero")]["cvar"] # the tail became a gain
|
||||||
|
assert risk[(0.8, "model")]["cvar"] < risk[(0.8, "zero")]["cvar"]
|
||||||
|
# the drift enters multiplicatively (e^{r+d} - 1), so the gap is well below mu * V
|
||||||
|
gap = risk[(1.5, "zero")]["cvar"] - risk[(1.5, "model")]["cvar"]
|
||||||
|
assert 0.3 * 1.5e6 < gap < 1.5e6
|
||||||
|
|
||||||
|
res = svc.optimize(OptimizationRequest(**{**base, "expected_returns": "historical"}))
|
||||||
|
if res.cvar <= 0: # the historical forecast is large enough to make the limit vacuous
|
||||||
|
assert any("constrains nothing" in w for w in res.warnings)
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_all_cash_utility_solution_says_why(md, settings):
|
||||||
|
"""objective='utility' with a large lambda is optimal in cash -- which must not look like a bug."""
|
||||||
|
from quantcore.optimization.service import OptimizationRequest, OptimizationService
|
||||||
|
|
||||||
|
svc = OptimizationService(settings, md)
|
||||||
|
common = dict(cash=1_000_000.0, max_assets=6, horizon="week", scenario_model="gaussian_exact",
|
||||||
|
solver="highs", time_limit=120, objective="utility")
|
||||||
|
big = svc.optimize(OptimizationRequest(risk_aversion=50.0, **common))
|
||||||
|
assert big.status in ("Optimal", "FeasibleFound") and big.invested == 0
|
||||||
|
assert any("nothing was invested" in w and "risk_aversion" in w for w in big.warnings)
|
||||||
|
small = svc.optimize(OptimizationRequest(risk_aversion=0.0, **common))
|
||||||
|
assert small.invested > 0 and not any("nothing was invested" in w for w in small.warnings)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_weighted_cvar_objective_is_routed_to_highs_and_says_so(md, settings):
|
||||||
|
"""cuOpt aborts when the CVaR is the objective *and* the scenarios carry probabilities.
|
||||||
|
|
||||||
|
The routing is that narrow on purpose: the same objective with uniform weights solves on the GPU,
|
||||||
|
so only the weighted case may be diverted, and only under solver='auto'.
|
||||||
|
"""
|
||||||
|
from quantcore.optimization.service import OptimizationRequest, OptimizationService
|
||||||
|
|
||||||
|
svc = OptimizationService(settings, md)
|
||||||
|
common = dict(cash=1_000_000.0, max_assets=6, horizon="week", scenario_model="gbm",
|
||||||
|
n_scenarios=1024, solver="auto", time_limit=120, risk_drift="zero")
|
||||||
|
weighted = svc.optimize(OptimizationRequest(objective="min_cvar", target_return=0.0005,
|
||||||
|
importance_sampling="market", **common))
|
||||||
|
assert weighted.solver == "highs"
|
||||||
|
assert any("probability-weighted" in w for w in weighted.warnings)
|
||||||
|
|
||||||
|
# uniform weights: no diversion (cuOpt is unreachable in the tests, so it still lands on HiGHS --
|
||||||
|
# through the fallback, which says something different)
|
||||||
|
plain = svc.optimize(OptimizationRequest(objective="min_cvar", target_return=0.0005,
|
||||||
|
importance_sampling="off", **common))
|
||||||
|
assert not any("probability-weighted" in w for w in plain.warnings)
|
||||||
|
# the constrained form is never diverted either
|
||||||
|
other = svc.optimize(OptimizationRequest(objective="max_return", cvar_limit_fraction=0.05,
|
||||||
|
importance_sampling="market", **common))
|
||||||
|
assert not any("probability-weighted" in w for w in other.warnings)
|
||||||
197
tests/test_simulation.py
Normal file
197
tests/test_simulation.py
Normal file
@ -0,0 +1,197 @@
|
|||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from quantcore.models.covariance import CovarianceModel, cholesky_factor, symmetric_inv_sqrt, symmetric_sqrt
|
||||||
|
from quantcore.models.simulation import EmpiricalInnovationSimulator, GBMSimulator
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def r():
|
||||||
|
rng = np.random.default_rng(7)
|
||||||
|
L = np.array([[1.0, 0, 0], [0.6, 0.8, 0], [0.3, 0.2, 0.9]])
|
||||||
|
x = rng.standard_t(6, size=(1500, 3)) @ L.T * 0.01
|
||||||
|
return pd.DataFrame(x, index=pd.bdate_range("2019-01-01", periods=1500), columns=list("ABC"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_matrix_roots(r):
|
||||||
|
cov = r.cov().values
|
||||||
|
s = symmetric_sqrt(cov)
|
||||||
|
np.testing.assert_allclose(s @ s, cov, atol=1e-12)
|
||||||
|
np.testing.assert_allclose(symmetric_inv_sqrt(cov) @ s, np.eye(3), atol=1e-8)
|
||||||
|
L = cholesky_factor(cov)
|
||||||
|
np.testing.assert_allclose(L @ L.T, cov, atol=1e-12)
|
||||||
|
|
||||||
|
|
||||||
|
def test_gbm_scenarios_have_target_moments(r):
|
||||||
|
cm = CovarianceModel("rolling", window=500)
|
||||||
|
mu = pd.Series([0.001, 0.0, -0.001], index=list("ABC"))
|
||||||
|
scen = GBMSimulator(cm).simulate(r, n_scenarios=100_000, horizon_periods=5, mu=mu, seed=1)
|
||||||
|
target = cm.estimate(r).values * 5
|
||||||
|
# Sobol' sets are rounded up to a power of two; moment matching makes the covariance exact
|
||||||
|
assert scen.model == "gbm" and scen.n_scenarios == 131_072 and scen.horizon_periods == 5
|
||||||
|
assert scen.extra["n_requested"] == 100_000 and scen.extra["sampling"] == "sobol"
|
||||||
|
np.testing.assert_allclose(np.cov(scen.log_returns, rowvar=False, ddof=0), target, rtol=1e-8, atol=1e-14)
|
||||||
|
# arithmetic drift convention: the expected simple return over the horizon is mu * h (Ito correction)
|
||||||
|
np.testing.assert_allclose(scen.expected_simple, mu.values * 5)
|
||||||
|
np.testing.assert_allclose(scen.mean_simple_returns(), mu.values * 5, atol=3e-5)
|
||||||
|
# log convention: the log-drift is mu * h exactly
|
||||||
|
scen_log = GBMSimulator(cm).simulate(r, n_scenarios=4096, horizon_periods=5, mu=mu, seed=1, drift_convention="log")
|
||||||
|
np.testing.assert_allclose(scen_log.log_returns.mean(axis=0), mu.values * 5, atol=1e-12)
|
||||||
|
assert (scen_log.expected_simple > np.expm1(mu.values * 5)).all() # sigma^2/2 bonus under the log convention
|
||||||
|
# plain Monte Carlo without moment matching: the old behaviour, moments only approximately right
|
||||||
|
mc = GBMSimulator(cm).simulate(r, n_scenarios=100_000, horizon_periods=5, mu=mu, seed=1, sampling="mc",
|
||||||
|
moment_match=False)
|
||||||
|
assert mc.n_scenarios == 100_000
|
||||||
|
np.testing.assert_allclose(np.cov(mc.log_returns, rowvar=False), target, rtol=0.05, atol=1e-6)
|
||||||
|
|
||||||
|
|
||||||
|
def test_innovations_are_standardised(r):
|
||||||
|
sim = EmpiricalInnovationSimulator(CovarianceModel("ewm", lam=0.97))
|
||||||
|
eps = sim.innovations(r)
|
||||||
|
assert eps.shape[1] == 3 and len(eps) > 1000
|
||||||
|
cov = np.cov(eps.values, rowvar=False)
|
||||||
|
np.testing.assert_allclose(cov, np.eye(3), atol=0.15)
|
||||||
|
|
||||||
|
|
||||||
|
def test_empirical_scenarios_match_latest_covariance(r):
|
||||||
|
cm = CovarianceModel("ewm", lam=0.97)
|
||||||
|
sim = EmpiricalInnovationSimulator(cm)
|
||||||
|
scen = sim.simulate(r, n_scenarios=60_000, horizon_periods=3, seed=2, sampling="mc", moment_match=False)
|
||||||
|
target = cm.estimate(r).values * 3
|
||||||
|
np.testing.assert_allclose(np.cov(scen.log_returns, rowvar=False), target, rtol=0.2, atol=2e-6)
|
||||||
|
# zero expected return: the log-drift is the Ito correction, the mean simple return ~ 0
|
||||||
|
np.testing.assert_allclose(scen.mean_simple_returns(), 0.0, atol=5e-4)
|
||||||
|
exact = sim.simulate(r, n_scenarios=60_000, horizon_periods=3, seed=2) # Sobol' ranks + moment matching
|
||||||
|
assert exact.n_scenarios == 65_536
|
||||||
|
np.testing.assert_allclose(np.cov(exact.log_returns, rowvar=False, ddof=0), target, rtol=1e-8, atol=1e-14)
|
||||||
|
# fat tails survive the filter: kurtosis of the innovations above the Gaussian 0
|
||||||
|
from scipy.stats import kurtosis
|
||||||
|
assert kurtosis(sim.innovations(r).values).mean() > 0.3
|
||||||
|
|
||||||
|
|
||||||
|
def test_cholesky_root_variant(r):
|
||||||
|
sim = EmpiricalInnovationSimulator(CovarianceModel("rolling", window=250), root="cholesky")
|
||||||
|
eps = sim.innovations(r)
|
||||||
|
np.testing.assert_allclose(np.cov(eps.values, rowvar=False), np.eye(3), atol=0.15)
|
||||||
|
scen = sim.simulate(r, n_scenarios=5000, horizon_periods=1, seed=3, sampling="mc")
|
||||||
|
assert scen.extra["root"] == "cholesky" and scen.log_returns.shape == (5000, 3)
|
||||||
|
|
||||||
|
|
||||||
|
def test_restandardisation_reproduces_model_covariance(r):
|
||||||
|
cm = CovarianceModel("ewm", lam=0.97, shrinkage=0.0)
|
||||||
|
for mode in ("diagonal", "full"):
|
||||||
|
sim = EmpiricalInnovationSimulator(cm, standardize=mode)
|
||||||
|
scen = sim.simulate(r, n_scenarios=80_000, horizon_periods=1, seed=5, sampling="mc", moment_match=False)
|
||||||
|
model = cm.estimate(r).values
|
||||||
|
sc = np.cov(scen.log_returns, rowvar=False)
|
||||||
|
np.testing.assert_allclose(np.sqrt(np.diag(sc)), np.sqrt(np.diag(model)), rtol=0.03)
|
||||||
|
assert scen.extra["standardize"] == mode and scen.extra["innovation_variance_median"] > 0
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
EmpiricalInnovationSimulator(cm, standardize="bogus")
|
||||||
|
|
||||||
|
|
||||||
|
def test_auto_shrinkage_intensity_properties():
|
||||||
|
from quantcore.measurement.estimators import shrinkage_intensity
|
||||||
|
|
||||||
|
rng = np.random.default_rng(11)
|
||||||
|
assert shrinkage_intensity(rng.standard_normal((80, 40))) == 1.0 # independent -> full shrinkage
|
||||||
|
L = np.linalg.cholesky(0.4 * np.eye(10) + 0.6)
|
||||||
|
strong = shrinkage_intensity(rng.standard_normal((2000, 10)) @ L.T)
|
||||||
|
assert 0.0 <= strong < 0.05 # strongly correlated, long sample -> little shrinkage
|
||||||
|
cm = CovarianceModel("rolling", window=100, shrinkage="auto")
|
||||||
|
x = pd.DataFrame(rng.standard_normal((300, 20)) @ np.linalg.cholesky(0.5 * np.eye(20) + 0.5).T)
|
||||||
|
assert 0 < cm.intensity(x) < 1 and cm.describe()["shrinkage"] == "auto"
|
||||||
|
|
||||||
|
|
||||||
|
def test_importance_sampling_tilt_both_models(r):
|
||||||
|
|
||||||
|
from quantcore.models.simulation import TiltSpec
|
||||||
|
|
||||||
|
cm = CovarianceModel("ewm", lam=0.97)
|
||||||
|
exposure = np.array([30_000.0, 50_000.0, 20_000.0])
|
||||||
|
beta = 0.99
|
||||||
|
for sim in (GBMSimulator(cm), EmpiricalInnovationSimulator(cm)):
|
||||||
|
plain = sim.simulate(r, n_scenarios=4096, horizon_periods=5, seed=9)
|
||||||
|
tilted = sim.simulate(r, n_scenarios=4096, horizon_periods=5, seed=9,
|
||||||
|
tilt=TiltSpec.at_confidence(exposure, beta, mixture=0.5))
|
||||||
|
assert plain.probabilities is None and tilted.probabilities is not None
|
||||||
|
assert abs(tilted.probabilities.sum() - 1) < 1e-12 and tilted.describe()["effective_sample_size"] > 2000
|
||||||
|
# exact first two moments in both cases (weighted for the tilted set)
|
||||||
|
w = tilted.weights
|
||||||
|
np.testing.assert_allclose(w @ tilted.log_returns, plain.log_returns.mean(axis=0), atol=1e-12)
|
||||||
|
cov_t = (tilted.log_returns - w @ tilted.log_returns).T @ ((tilted.log_returns - w @ tilted.log_returns) * w[:, None])
|
||||||
|
np.testing.assert_allclose(cov_t, np.cov(plain.log_returns, rowvar=False, ddof=0), rtol=1e-8, atol=1e-14)
|
||||||
|
# many more scenarios beyond the reference portfolio's VaR, with the right total probability
|
||||||
|
loss_p = -(plain.simple_returns() @ exposure)
|
||||||
|
loss_t = -(tilted.simple_returns() @ exposure)
|
||||||
|
var = np.quantile(loss_p, beta)
|
||||||
|
assert (loss_t > var).sum() > 5 * (loss_p > var).sum()
|
||||||
|
assert abs(w[loss_t > var].sum() - (1 - beta)) < 0.01
|
||||||
|
assert "importance_sampling" in tilted.extra and tilted.hash != plain.hash
|
||||||
|
# the default tilt mixes z_0.95 and z_0.99 blocks, so one set serves both levels
|
||||||
|
both = sim.simulate(r, n_scenarios=4096, horizon_periods=5, seed=9, tilt=TiltSpec(exposure=exposure))
|
||||||
|
w2 = both.weights
|
||||||
|
loss_b = -(both.simple_returns() @ exposure)
|
||||||
|
for level in (0.95, 0.99):
|
||||||
|
var_l = np.quantile(loss_p, level)
|
||||||
|
assert (loss_b > var_l).sum() > 3 * (loss_p > var_l).sum()
|
||||||
|
assert abs(w2[loss_b > var_l].sum() - (1 - level)) < 0.01
|
||||||
|
# determinism: same seed -> same hash, other seed -> different
|
||||||
|
a = GBMSimulator(cm).simulate(r, n_scenarios=512, seed=1)
|
||||||
|
b = GBMSimulator(cm).simulate(r, n_scenarios=512, seed=1)
|
||||||
|
c = GBMSimulator(cm).simulate(r, n_scenarios=512, seed=2)
|
||||||
|
assert a.hash == b.hash != c.hash
|
||||||
|
|
||||||
|
|
||||||
|
def test_scenario_distribution_reports_the_model_quantities(r):
|
||||||
|
sim = GBMSimulator(CovarianceModel("ewm", lam=0.97, shrinkage=0.0))
|
||||||
|
scen = sim.simulate(r, n_scenarios=8192, horizon_periods=5, mu=pd.Series(0.0004, index=r.columns), seed=5)
|
||||||
|
v = np.array([400_000.0, 300_000.0, 300_000.0])
|
||||||
|
d = scen.distribution(v, bins=40, confidence=(0.95, 0.99))
|
||||||
|
cov_h = CovarianceModel("ewm", lam=0.97, shrinkage=0.0).estimate(r).values * 5
|
||||||
|
sigma = np.sqrt(v @ cov_h @ v)
|
||||||
|
assert d["volatility"] == pytest.approx(sigma, rel=0.02)
|
||||||
|
# the reported mean is the mean of the *displayed* sample; moment matching is exact in log
|
||||||
|
# space, so it equals the model's expectation only up to the convexity of exp
|
||||||
|
assert d["expected_pnl"] == pytest.approx(float(scen.expected_simple @ v), rel=1e-4)
|
||||||
|
assert sum(d["histogram"]["mass"]) == pytest.approx(1.0)
|
||||||
|
assert len(d["histogram"]["edges"]) == 41
|
||||||
|
# Gaussian reference: VaR = z sigma - mean, CVaR = phi(z)/(1-beta) sigma - mean
|
||||||
|
from scipy.stats import norm
|
||||||
|
for beta in (0.95, 0.99):
|
||||||
|
z = norm.ppf(beta)
|
||||||
|
assert d["risk"][f"{beta:g}"]["var"] == pytest.approx(z * sigma - d["expected_pnl"], rel=0.03)
|
||||||
|
assert d["risk"][f"{beta:g}"]["cvar"] == pytest.approx(
|
||||||
|
norm.pdf(z) / (1 - beta) * sigma - d["expected_pnl"], rel=0.03)
|
||||||
|
assert abs(d["skewness"]) < 0.25 and d["kurtosis"] == pytest.approx(3.0, abs=0.4)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_tilted_set_reports_the_same_distribution_as_an_untilted_one(r):
|
||||||
|
from quantcore.models.simulation import TiltSpec
|
||||||
|
sim = GBMSimulator(CovarianceModel("ewm", lam=0.97, shrinkage=0.0))
|
||||||
|
v = np.array([400_000.0, 300_000.0, 300_000.0])
|
||||||
|
kw = dict(n_scenarios=8192, horizon_periods=5, mu=pd.Series(0.0004, index=r.columns), seed=11)
|
||||||
|
plain = sim.simulate(r, **kw).distribution(v)
|
||||||
|
tilted = sim.simulate(r, **kw, tilt=TiltSpec(exposure=v)).distribution(v)
|
||||||
|
assert tilted["weighted"] and tilted["effective_sample_size"] < 8192
|
||||||
|
assert tilted["expected_pnl"] == pytest.approx(plain["expected_pnl"], rel=1e-6)
|
||||||
|
assert tilted["volatility"] == pytest.approx(plain["volatility"], rel=0.02)
|
||||||
|
assert tilted["risk"]["0.99"]["cvar"] == pytest.approx(plain["risk"]["0.99"]["cvar"], rel=0.03)
|
||||||
|
for q in ("0.05", "0.5"):
|
||||||
|
assert tilted["quantiles"][q] == pytest.approx(plain["quantiles"][q], rel=0.05)
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_reports_price_returns_that_cannot_fall_below_minus_one(r):
|
||||||
|
"""A long horizon makes the two spaces look very different: a log-return of -2 is -86 % of the
|
||||||
|
price. The table must carry both, and the price column must be bounded below by -100 %."""
|
||||||
|
sim = GBMSimulator(CovarianceModel("ewm", lam=0.97, shrinkage=0.0))
|
||||||
|
scen = sim.simulate(r, n_scenarios=4096, horizon_periods=63, mu=pd.Series(0.0004, index=r.columns), seed=3)
|
||||||
|
t = scen.summary()
|
||||||
|
for col in ("q01", "q05", "q50", "q95", "min", "max"):
|
||||||
|
np.testing.assert_allclose(t[f"{col}_simple"], np.expm1(t[col]), rtol=1e-12) # a monotone map
|
||||||
|
assert (t["min_simple"] > -1).all() and (t["min"] < t["min_simple"]).all()
|
||||||
|
assert (t["max_simple"] > t["max"]).all()
|
||||||
|
# the scenario mean in price space is above the log mean (Jensen) and matches the model's
|
||||||
|
assert (t["mean_simple"] > t["mean"]).all()
|
||||||
|
assert np.allclose(t["mean_simple"], t["expected_simple"], rtol=0.05)
|
||||||
165
tests/test_solvers.py
Normal file
165
tests/test_solvers.py
Normal file
@ -0,0 +1,165 @@
|
|||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
import scipy.sparse as sp
|
||||||
|
|
||||||
|
from quantcore.config import Settings
|
||||||
|
from quantcore.optimization.problem import MILPModel
|
||||||
|
from quantcore.optimization.solvers import CuOptSolver, HighsSolver, solve_with_fallback, to_cuopt_payload
|
||||||
|
|
||||||
|
|
||||||
|
def readme_model() -> MILPModel:
|
||||||
|
# maximise 3x + 2y s.t. 2x + 3y <= 12, 4x + y <= 10, x,y >= 0 integer -> x = 2, y = 2, obj 10
|
||||||
|
A = sp.csr_matrix(np.array([[2.0, 3.0], [4.0, 1.0]]))
|
||||||
|
return MILPModel(c=np.array([3.0, 2.0]), A=A, row_lb=np.array([-np.inf, -np.inf]), row_ub=np.array([12.0, 10.0]),
|
||||||
|
var_lb=np.zeros(2), var_ub=np.array([np.inf, np.inf]), var_types=["I", "I"], var_names=["x", "y"],
|
||||||
|
row_names=["c1", "c2"], maximize=True)
|
||||||
|
|
||||||
|
|
||||||
|
def test_payload_format():
|
||||||
|
payload = to_cuopt_payload(readme_model(), time_limit=5)
|
||||||
|
assert payload["csr_constraint_matrix"] == {"offsets": [0, 2, 4], "indices": [0, 1, 0, 1], "values": [2.0, 3.0, 4.0, 1.0]}
|
||||||
|
assert payload["constraint_bounds"]["lower_bounds"] == ["ninf", "ninf"]
|
||||||
|
assert payload["variable_bounds"]["upper_bounds"] == ["inf", "inf"]
|
||||||
|
assert payload["variable_types"] == ["I", "I"] and payload["maximize"] is True
|
||||||
|
assert payload["solver_config"]["time_limit"] == 5.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_highs_solves_readme_example():
|
||||||
|
res = HighsSolver().solve(readme_model())
|
||||||
|
assert res.ok and res.status == "Optimal"
|
||||||
|
np.testing.assert_allclose(res.x, [2.0, 2.0])
|
||||||
|
assert res.objective == pytest.approx(10.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_fallback_when_cuopt_unreachable():
|
||||||
|
settings = Settings(cuopt_url="http://127.0.0.1:9", solver_fallback=True, cuopt_request_timeout=2)
|
||||||
|
res = solve_with_fallback(readme_model(), settings, solver="auto")
|
||||||
|
assert res.solver == "highs" and "fallback" in res.message and res.objective == pytest.approx(10.0)
|
||||||
|
with pytest.raises(ConnectionError):
|
||||||
|
solve_with_fallback(readme_model(), settings, solver="cuopt")
|
||||||
|
assert CuOptSolver(settings).health()["reachable"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_fallback_when_cuopt_dies_during_the_solve(monkeypatch):
|
||||||
|
"""cuOpt accepted the job, then its server went away while the client polled for the solution
|
||||||
|
(seen with a read-only container whose solver process crashed): 'auto' must still answer, on HiGHS."""
|
||||||
|
import requests
|
||||||
|
|
||||||
|
from quantcore.optimization import solvers as S
|
||||||
|
|
||||||
|
settings = Settings(cuopt_url="http://127.0.0.1:9", solver_fallback=True, cuopt_poll_interval=0.01)
|
||||||
|
monkeypatch.setattr(S.CuOptSolver, "reachable", lambda self, ttl=30.0: True)
|
||||||
|
monkeypatch.setattr(S.CuOptSolver, "_post_with_retry", lambda self, payload: "job-1")
|
||||||
|
|
||||||
|
def lost(*args, **kwargs):
|
||||||
|
raise requests.ConnectionError("Max retries exceeded")
|
||||||
|
|
||||||
|
monkeypatch.setattr(S.requests, "get", lost)
|
||||||
|
monkeypatch.setattr(S.requests, "delete", lost)
|
||||||
|
res = solve_with_fallback(readme_model(), settings, solver="auto")
|
||||||
|
assert res.solver == "highs" and "connection lost" in res.message and res.objective == pytest.approx(10.0)
|
||||||
|
with pytest.raises(ConnectionError):
|
||||||
|
solve_with_fallback(readme_model(), settings, solver="cuopt")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not CuOptSolver(Settings()).health().get("reachable"), reason="cuOpt server not reachable")
|
||||||
|
def test_cuopt_solves_readme_example():
|
||||||
|
res = CuOptSolver(Settings()).solve(readme_model(), time_limit=10)
|
||||||
|
assert res.ok and res.solver == "cuopt"
|
||||||
|
np.testing.assert_allclose(res.x, [2.0, 2.0])
|
||||||
|
assert res.objective == pytest.approx(10.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_cuopt_retries_down_the_method_ladder_after_an_abort(monkeypatch):
|
||||||
|
"""cuOpt's default method aborts on some CVaR-objective models; the client works down
|
||||||
|
``CUOPT_RETRY_METHODS`` and reports which method finally solved it."""
|
||||||
|
from quantcore.optimization import solvers as S
|
||||||
|
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
class Response:
|
||||||
|
def __init__(self, payload, status=200):
|
||||||
|
self._payload = payload
|
||||||
|
self.status_code = status
|
||||||
|
self.text = str(payload)
|
||||||
|
|
||||||
|
def json(self):
|
||||||
|
return self._payload
|
||||||
|
|
||||||
|
def raise_for_status(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def fake_post(url, data=None, headers=None, timeout=None, **kw): # noqa: ARG001
|
||||||
|
import msgpack
|
||||||
|
|
||||||
|
calls.append(msgpack.unpackb(data)["solver_config"])
|
||||||
|
return Response({"reqId": f"req{len(calls)}"})
|
||||||
|
|
||||||
|
def fake_get(url, headers=None, timeout=None, **kw):
|
||||||
|
if len(calls) == 1: # first attempt: aborted
|
||||||
|
return Response({"error_result": True, "error": "req1 was aborted"})
|
||||||
|
return Response({"response": {"solver_response": {
|
||||||
|
"status": "Optimal",
|
||||||
|
"solution": {"primal_solution": [1.0, 0.0], "primal_objective": 1.0, "solver_time": 0.1,
|
||||||
|
"milp_statistics": {"mip_gap": 0.0}},
|
||||||
|
}}})
|
||||||
|
|
||||||
|
class FakeSession: # the client opens a fresh session per submit attempt
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *exc):
|
||||||
|
return False
|
||||||
|
|
||||||
|
post = staticmethod(fake_post)
|
||||||
|
|
||||||
|
monkeypatch.setattr(S.requests, "Session", FakeSession)
|
||||||
|
monkeypatch.setattr(S.requests, "get", fake_get)
|
||||||
|
monkeypatch.setattr(S.requests, "delete", lambda *a, **k: Response({}))
|
||||||
|
|
||||||
|
model = readme_model()
|
||||||
|
res = S.CuOptSolver(Settings(cuopt_url="http://example.invalid")).solve(model, time_limit=5)
|
||||||
|
first = S.CUOPT_RETRY_METHODS[0]
|
||||||
|
assert res.status == "Optimal" and f"method={first}" in res.message
|
||||||
|
assert calls[0].get("method") is None and calls[1]["method"] == first # default, then the ladder
|
||||||
|
|
||||||
|
|
||||||
|
def test_cuopt_reports_when_every_method_aborts(monkeypatch):
|
||||||
|
"""All rungs exhausted: the caller is told that, not just that the first attempt aborted."""
|
||||||
|
from quantcore.optimization import solvers as S
|
||||||
|
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
class Response:
|
||||||
|
def __init__(self, payload):
|
||||||
|
self._payload, self.status_code, self.text = payload, 200, str(payload)
|
||||||
|
|
||||||
|
def json(self):
|
||||||
|
return self._payload
|
||||||
|
|
||||||
|
def raise_for_status(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
class FakeSession:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *exc):
|
||||||
|
return False
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def post(url, data=None, headers=None, timeout=None, **kw):
|
||||||
|
import msgpack
|
||||||
|
|
||||||
|
calls.append(msgpack.unpackb(data)["solver_config"])
|
||||||
|
return Response({"reqId": f"req{len(calls)}"})
|
||||||
|
|
||||||
|
monkeypatch.setattr(S.requests, "Session", FakeSession)
|
||||||
|
monkeypatch.setattr(S.requests, "get",
|
||||||
|
lambda *a, **k: Response({"error_result": True, "error": "req was aborted"}))
|
||||||
|
monkeypatch.setattr(S.requests, "delete", lambda *a, **k: Response({}))
|
||||||
|
|
||||||
|
res = S.CuOptSolver(Settings(cuopt_url="http://example.invalid")).solve(readme_model(), time_limit=5)
|
||||||
|
assert res.status == "Error"
|
||||||
|
assert str(S.CUOPT_RETRY_METHODS) in res.message
|
||||||
|
assert len(calls) == 1 + len(S.CUOPT_RETRY_METHODS) # the default plus every rung
|
||||||
Loading…
x
Reference in New Issue
Block a user